Automatic voice processing method based on language large model and electronic equipment

Through the automated speech processing method based on language big model, LoRA technology and natural language processing technology are used to solve the problem of insufficient recognition of semantic models in multiple rounds of speech interactions, improve the accuracy and generalization ability of semantic understanding, and realize automation and Real-time, providing a better user experience.

CN119091864APending Publication Date: 2024-12-06CHINA FAW CO LTD +1
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411199535.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

It is difficult for the existing technology to achieve automated and intelligent semantic understanding, especially in multiple rounds of speech interaction scenarios. The semantic model lacks relevant samples during training, resulting in insufficient semantic recognition and the inability to accurately understand user intentions.

Method used

Using an automated speech processing method based on language big models, a multi-round dialogue data is formed by obtaining user voice interaction data, and the language big model is fine-tuned by LoRA technology, a low-rank matrix is ​​introduced to adjust the model parameters, and combined with natural language processing and inverse Propagation algorithms, optimize models to improve semantic recognition capabilities.

Benefits of technology

It significantly improves the accuracy of semantic understanding and model generalization capabilities, reduces the cost of manual labeling, realizes self-iteration optimization and real-time improvement of semantic models, and provides a more accurate user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091864A_ABST
    Figure CN119091864A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic voice processing method based on a large language model and electronic equipment. The method comprises the steps of obtaining user voice interaction data, forming a multi-round dialogue data set, performing natural language processing on the dialogue data set by using the large language model, performing fine tuning training on the multi-round dialogue data set in a labeling manner, and generating training set data; according to the training set data, a LoRA technology is adopted to carry out fine tuning on a language large model, a fine-tuned large language model is formed, based on the fine-tuned large language model, semantic recognition is carried out on the multi-round dialogue data set, and corresponding correction intention data is generated. According to the method, the development cost is reduced, the real-time performance is improved, the large model is used for recognizing the real negative feedback case of the user and then correcting the real negative feedback case, the cost is much lower than that of manual recognition and labeling, the speed is high, meanwhile, the real-time performance can be greatly improved, and the voice experience of the user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an automatic speech processing method and electronic equipment, and in particular to an automatic speech processing method and electronic equipment based on a large language model. Background Art

[0002] Voice is an important interactive channel in the process of automotive intelligence. In the face of different user requirements and statements, voice is needed to understand user intentions more accurately, so enhancing semantic understanding capabilities is the key. For example, in multiple rounds of user voice interaction cases, some cases can correctly understand user intentions, but some cases cannot correctly identify user semantics. As for why the semantics are not identified, there may be many reasons. The most critical one is that the semantic model did not add samples of these statements during the training process, or there were only a few similar samples during training, which were not enough for the model to learn these semantics.

[0003] The existing technology method is to enhance semantic generalization and understanding. This method requires the enhancement of high-quality data samples, but mining high-quality data samples and supplementing them into the model is a relatively difficult problem. Under normal circumstances, it is necessary to find these cases and manually annotate them, but it is difficult to achieve automated and intelligent processing methods. The above situation can no longer meet people's requirements and needs to be improved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide an automated speech processing method and electronic device based on a large language model, which uses a large language model to increase semantic understanding and generalization capabilities, accurately identify user semantics to improve user experience, and solve the shortcomings of the prior art.

[0005] The present invention provides the following scheme:

[0006] An automatic speech processing method based on a large language model is applied to an automatic speech processing system based on a large language model, comprising:

[0007] Obtain user voice interaction data to form a multi-round dialogue data set, use the language model to perform natural language processing on the dialogue data set, and fine-tune the multi-round dialogue data set through annotation to generate training set data;

[0008] Fine-tune the language model using LoRA technology according to the training set data to form a fine-tuned large language model, wherein the fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix into each level of the language model, adjusting the model parameters, and fine-tuning the preprocessed training set data;

[0009] Based on the fine-tuned large language model, semantic recognition is performed on the multi-round dialogue dataset to generate corresponding correction intention data.

[0010] Furthermore, the acquiring of user voice interaction data to form a multi-round dialogue data set, performing natural language processing on the dialogue data set using a large language model, and fine-tuning the multi-round dialogue data set by annotation to generate training set data further includes:

[0011] Acquire the voice interaction data between the user and the intelligent system, and process it through the context retention algorithm to form a multi-round dialogue data set. The context retention algorithm is specifically: sequence model and attention mechanism algorithm;

[0012] Label the multi-round dialogue dataset, and perform natural language processing on the labeled multi-round dialogue dataset based on the language big model to obtain natural language understanding data for identifying user intent and evaluating dialogue satisfaction;

[0013] Fine-tune the natural language understanding data, adjust the parameters of the large language model through the back-propagation algorithm and the gradient descent optimization method, calculate the gradient of the large language model parameters through the loss function, and use the Adam or SGD optimization algorithm to update the parameters, and perform iterative training to minimize the difference between the prediction results of the large language model and the actual annotations.

[0014] Furthermore, the method of fine-tuning the large language model using the LoRA technology according to the training set data to form a fine-tuned large language model further includes:

[0016] Determine the trainable low-rank matrix of each layer in the fine-tuned large language model, and use the data in the training set to calculate the gradient of the low-rank matrix through the back-propagation algorithm;

[0017] According to the gradient update rule, the low-rank matrix is ​​updated to optimize the fit of the model to the training set;

[0018] Vectorize the user voice data and convert it into numerical features suitable for model input.

[0019] Furthermore, the vectorization processing of the user voice data is specifically: denoising and normalizing the user voice data, and applying Fourier transform to convert the voice signal into frequency domain data;

[0020] The principal component analysis is performed on the feature vector obtained after the vectorization process to reduce the dimension, and the representative feature vector is extracted, and the representative feature vector is standardized.

[0021] Furthermore, the method of fine-tuning the large language model using the LoRA technology according to the training set data to form a fine-tuned large language model further includes:

[0022] Modify the weight matrix of the large language model and adjust the model parameters by multiplying the weight matrix with the low-rank matrix;

[0023] Decompose the parameter matrix of each level of the model, and use the singular value decomposition (SVD) algorithm to initialize the low-rank matrix of each level to capture the change direction of the parameter matrix;

[0024] The back-propagation algorithm is used to calculate the gradient of each low-rank matrix relative to the loss function, and the loss function is derived by the chain rule to obtain the gradient information of each low-rank matrix element.

[0025] Furthermore, the weight matrix of the modified large language model is specifically:

[0026] Analyze the weight matrix in the large language model to determine the relationship between the weight matrix and the model performance;

[0027] Construct a low-rank matrix that matches the dimension of the weight matrix, and multiply the weight matrix with the low-rank matrix through multiplication operation to generate an adjusted weight matrix;

[0028] Using the training set data, the gradient of the adjusted weight matrix is ​​calculated through the back propagation algorithm, and the low-rank matrix is ​​updated using the optimization algorithm based on the calculated gradient.

[0029] An automated speech processing system based on a large language model, used to implement the automated speech method based on a large language model, comprising:

[0030] The user voice data acquisition module is used to acquire user voice interaction data to form a multi-round dialogue data set, perform natural language processing on the dialogue data set using a large language model, and fine-tune the multi-round dialogue data set by annotation to generate training set data;

[0031] A language model fine-tuning module is used to fine-tune the language model using LoRA technology according to the training set data to form a fine-tuned large language model. The fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix into each level of the language model, adjusting the model parameters, and fine-tuning the preprocessed training set data;

[0032] The correction intention data generation module is used to perform semantic recognition on the multi-round dialogue data set based on the fine-tuned large language model to generate corresponding correction intention data.

[0033] A smart cockpit is provided with the automatic speech processing system based on the language large model, and executes the automatic speech processing method based on the language large model.

[0034] An electronic device comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0035] A computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0036] Compared with the prior art, the present invention has the following advantages:

[0038] The present invention improves the accuracy of semantic understanding, and judges whether the semantics satisfy the user based on the actual user data based on the big model, which greatly improves the judgment accuracy. It can accurately find the deficiencies of the semantic model and correct them with the big model. Secondly, the samples corrected by the big model are output to the semantic model for learning, so that the semantic model can learn these fundamentally and reduce the cost of the manual labeling process. By using this model, the semantic model can learn more and broader cases of negative feedback from real users, continuously improve the accuracy of semantic understanding, and achieve a virtuous circle.

[0039] The present invention improves the generalization ability of the language model. After the large model digs out such semantically problematic cases and allows it to correct them, the large model is used to generate more similar cases for the corrected cases. Finally, these generalized cases are used as samples for the semantic model to learn, thereby achieving the ability to draw inferences from one example.

[0040] The present invention reduces development costs and improves real-time performance. It uses a large model to identify real negative feedback cases from users and then allows the large model to make corrections. This is much cheaper and faster than manual identification and labeling. It can also greatly improve real-time performance and enhance the user's experience of voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1It is a flow chart of an automated speech processing method based on a large language model.

[0043] Figure 1A It is a flow chart of the optimization technical solution of step S1.

[0044] Figure 1B It is a flow chart of the optimization technical solution of step S2.

[0045] Figure 1C This is a flowchart of step S2 using the LoRA technology to fine-tune the language model using the training set data.

[0046] Figure 1D It is a flow chart of the optimization technical solution of step S24.

[0047] Figure 2 This is the architecture diagram of the automated speech processing system based on the language big model.

[0049] Figure 3 It is a chart of multiple rounds of voice interaction cases of users.

[0050] Figure 4 It is an implementation method of the embodiment of the present invention in a specific application scenario.

[0051] Figure 5 It is a structural diagram of an electronic device. DETAILED DESCRIPTION

[0052] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 The automatic speech processing method based on the language large model is applied to the automatic speech processing system based on the language large model, including:

[0054] Step S1, obtaining user voice interaction data to form a multi-round dialogue data set, using a large language model to perform natural language processing on the dialogue data set, fine-tuning the multi-round dialogue data set by annotation, and generating training set data; in step S1, a multi-round dialogue data set is formed to retain rich contextual information, and training set data is generated to identify user satisfaction and the accuracy of voice intent.

[0055] Step S2, fine-tuning the large language model using LoRA technology according to the training set data to form a fine-tuned large language model, wherein the fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix inside each level of the large language model, adjusting the model parameters, and fine-tuning the preprocessed training set data; the purpose of step S2 is to reduce the number of parameters and resource requirements while maintaining model performance.

[0056] Step S3, based on the fine-tuned large language model, semantic recognition is performed on the multi-round dialogue data set to generate corresponding correction intention data. The purpose of step S3 is to predict and identify interaction points with insufficient semantic understanding, and enhance the user's semantic understanding and generalization capabilities.

[0057] The technical solution provided in steps S1 to S3 achieves efficient processing of user voice interaction data and enhanced semantic understanding by combining natural language processing technology and model fine-tuning strategies. It captures the coherence and complexity of user interactions through data set construction and preprocessing, and provides a high-quality data foundation for subsequent semantic analysis and model training. It uses LoRA technology to fine-tune the pre-trained language model, and effectively reduces the number of model parameters and resource requirements by introducing low-rank matrices at each level of the model, while maintaining the performance of the model, so that the model can more accurately identify user satisfaction and voice intent, thereby improving the accuracy of semantic understanding. Based on the fine-tuned large language model, in-depth semantic recognition is performed on multi-round dialogue data sets, correction intent data is generated, and interaction points with insufficient semantic understanding are predicted and identified. By providing correction intent, the user's semantic understanding and generalization capabilities are enhanced.

[0059] like Figure 1A As shown, in the optimization technical solution of step S1, the user voice interaction data is obtained to form a multi-round dialogue data set, the dialogue data set is subjected to natural language processing using a large language model, and the multi-round dialogue data set is fine-tuned and trained by annotation to generate training set data, further comprising:

[0060] Step S11, obtaining the voice interaction data between the user and the intelligent system, and processing it through a context-retention algorithm to form a multi-round dialogue data set. The context-retention algorithm is specifically: a sequence model, an attention mechanism algorithm, and the attention mechanism algorithm can be various algorithms based on the Transformer architecture.

[0061] Step S12, annotating the multi-round dialogue data set, performing natural language processing on the annotated multi-round dialogue data set based on the language big model, and obtaining natural language understanding data for identifying user intentions and evaluating dialogue satisfaction.

[0062] Step S13, fine-tuning training is performed on the natural language understanding data, the parameters of the large language model are adjusted by the back propagation algorithm and the gradient descent optimization method, the gradient of the large language model parameters is calculated by the loss function, and the parameters are updated by the Adam or SGD optimization algorithm, and iterative training is performed to minimize the difference between the prediction result of the large language model and the actual annotation.

[0063] In the optimization technical solution provided in step S11 to step S13, by acquiring the voice interaction data between the user and the intelligent system and applying the context-retaining algorithm (such as the sequence model and the attention mechanism algorithm based on the Transformer architecture), a multi-round dialogue data set containing rich context information is formed to ensure that the dialogue information in the data set is complete and coherent, and to provide high-quality input for subsequent natural language processing. Based on the manual annotation of the constructed multi-round dialogue data set, the pre-trained language large model then performs natural language processing on the annotated data to identify the user's intention and evaluate the satisfaction of the dialogue, and fine-tunes the natural language understanding data, and adjusts the parameters of the large language model through the back-propagation algorithm and the gradient descent optimization method. By calculating the gradient of the loss function and using optimization algorithms such as Adam or SGD to update the parameters, the model is iteratively trained to minimize the difference between the model prediction and the actual annotation.

[0065] The optimization technical solution provided in steps S11 to S13 improves the accuracy of semantic understanding. Through the context retention algorithm and natural language processing, the model can more accurately understand and identify the user's voice intention, thereby improving the accuracy of semantic understanding. It enhances the conversation satisfaction assessment. Through natural language processing and fine-tuning training on the annotated data, the model can more accurately assess the conversation satisfaction and provide users with a more satisfactory interactive experience. Through the back propagation algorithm and gradient descent optimization method, as well as optimization algorithms such as Adam or SGD, the parameters of the model are finely adjusted, thereby improving the overall performance and generalization ability of the model, and ultimately achieving automated iterative training, so that the model can continuously learn and optimize, constantly adapt to new data and scenarios, and achieve self-iteration and performance improvement.

[0066] like Figure 1B As shown, in the optimization technical solution of step S2, the large language model is fine-tuned using LoRA technology according to the training set data to form a fine-tuned large language model, further comprising:

[0067] Step S21, determining the trainable low-rank matrix of each level in the fine-tuned large language model, and using the data in the training set to calculate the gradient of the low-rank matrix through the back propagation algorithm;

[0068] Step S22, updating the low-rank matrix according to the gradient update rule to optimize the fit of the model to the training set;

[0069] Step S23, vectorizing the user voice data to convert the user voice data into numerical features suitable for model input.

[0070] The optimization technology provided in steps S21 to S23 effectively fine-tune the large language model to adapt to a specific training data set, determine the trainable low-rank matrix at each level of the large language model, and use the back-propagation algorithm to calculate its gradient, thereby achieving fine adjustment of the model parameters. According to the calculated gradient, the low-rank matrix is ​​updated according to the set update rules, and the model's fit to the training set data is optimized, thereby improving the accuracy and efficiency of the model. Finally, the user's voice data is converted into numerical features to make it suitable as the input of the model, ensuring that the model can process and understand the voice data, providing a basis for subsequent natural language processing.

[0071] Exemplarily, in step S23, the vectorization processing of the user voice data is specifically: denoising and normalizing the user voice data, and applying Fourier transform to convert the voice signal into frequency domain data;

[0072] The principal component analysis (PCA) is performed on the feature vector obtained after the vectorization process to extract the representative feature vector, and the representative feature vector is standardized.

[0073] In the specific implementation of step S23, the user voice data is converted into a numerical feature vector suitable for input into the machine learning model. The conversion process includes preprocessing the voice data, frequency domain conversion, feature extraction and dimensionality reduction, and final standardization processing. The quality of the voice data is improved by denoising and normalization processing, which improves the data quality and makes the model training more stable and effective. The frequency domain conversion and PCA dimensionality reduction methods are used to enable the model to focus on the most representative features in the voice signal, thereby improving the feature expression ability and the recognition accuracy of the model, reducing the computing resources required for model training, and improving the efficiency of model training and prediction.

[0074] PCA dimensionality reduction, also known as principal component analysis (Principal Component Analysis) dimensionality reduction, is a statistical method used to reduce the dimension of a data set while retaining the variability of the original data as much as possible. It is suitable for the fields of machine learning and data mining.

[0075] PCA works as follows:

[0076] Data standardization: Each feature in the data set is standardized so that the mean of each feature is 0 and the standard deviation is 1, which is used to ensure that each feature is treated fairly during the dimensionality reduction process.

[0077] Covariance matrix calculation: Calculate the covariance matrix of the data to determine the correlation between the various features in the data. The covariance matrix is ​​used to describe the linear relationship between the data features.

[0078] Eigenvalue and eigenvector calculation: Calculate the eigenvalues ​​and corresponding eigenvectors of the covariance matrix. The eigenvalues ​​represent the variance of the data in the direction of each eigenvector, while the eigenvectors define this direction.

[0079] Select principal components: According to the size of the eigenvalues, select the most important eigenvectors, which are called principal components. The usual selection criterion is that the cumulative contribution rate (i.e., the variance of the first few principal components and the proportion of the total variance) reaches a predetermined threshold (such as 95%).

[0080] Construct a new feature space: Project the original data onto the selected principal components to construct a new feature space. The dimension of this new feature space is lower than that of the original data set, but it still retains most of the information.

[0081] Reduced Dimensionality Data: Use the new feature space as the reduced dimensionality data and apply the above data to subsequent analysis or machine learning model training.

[0082] like Figure 1C As shown, in step S2, the large language model is fine-tuned using LoRA technology according to the training set data to form a fine-tuned large language model, further comprising:

[0083] Step S24, modifying the weight matrix of the large language model, and adjusting the model parameters by multiplying the weight matrix by the low-rank matrix;

[0084] Step S25, decomposing the parameter matrix of each level of the model, identifying its potential low-rank structure, and initializing the low-rank matrix of each level using the singular value decomposition (SVD) algorithm to capture the change direction of the parameter matrix;

[0085] Step S26, using the back propagation algorithm to calculate the gradient of each low-rank matrix relative to the loss function, and performing chain rule derivation on the loss function to obtain the gradient information of each low-rank matrix element.

[0086] In the optimization technical solution provided by steps S24 to S26, the parameter adjustment process of the large language model is optimized by the algorithm, and the weight matrix is ​​multiplied by the low-rank matrix to achieve fine adjustment of the parameters of the large language model, improve the adaptability and performance of the model, and perform the processing flow of parameter matrix decomposition and low-rank structure identification. The parameter matrix is ​​decomposed for each level of the model to identify its potential low-rank structure, and the low-rank matrix of each level is initialized by the singular value decomposition (SVD) algorithm to capture the main change direction of the parameter matrix. Finally, the back-propagation algorithm is used to calculate the gradient of each low-rank matrix relative to the loss function, and gradient calculation and parameter update are performed. The gradient information is obtained by chain rule derivation to provide a basis for updating the model parameters.

[0087] The optimization technology solution provided in steps S24 to S26 achieves efficient optimization of large language model parameters by combining technical means such as weight matrix adjustment, low-rank structure recognition and gradient calculation, improves the performance and computational efficiency of the model, and provides a model foundation for building a high-performance natural language processing system.

[0088] like Figure 1D As shown, in step S24, the weight matrix of the large language model is modified as follows:

[0089] Step S241, analyzing the weight matrix in the large language model to determine the relationship between the weight matrix and the model performance;

[0090] Step S242, constructing a low-rank matrix that matches the dimension of the weight matrix, and multiplying the weight matrix by the low-rank matrix through a multiplication operation to generate an adjusted weight matrix;

[0091] Step S243, using the training set data, calculate the gradient of the adjusted weight matrix through the back propagation algorithm, and update the low-rank matrix using the optimization algorithm based on the calculated gradient.

[0092] In the technical solution provided in steps S241 to S243, by analyzing the relationship between the weight matrix and the model performance, the model parameters can be adjusted more specifically to optimize the model performance; the introduction of a low-rank matrix can significantly reduce the number of model parameters, reduce the complexity of the model and the computing resource requirements, while maintaining or improving the model performance; through multiplication operations and back-propagation algorithms, the gradient of the weight matrix can be efficiently calculated, and the low-rank matrix can be quickly updated, thereby improving the computational efficiency of model training and adjustment; the optimized weight matrix helps to improve the model's adaptability and generalization ability to new data, so that the model can perform better on different tasks and data sets; reducing the number of parameters and improving computing efficiency helps to reduce the cost of model training, including the consumption of time and hardware resources.

[0094] It can be seen that in the technical solution provided in steps S241 to S243, through the analysis of the weight matrix, the construction and adjustment of the low-rank matrix, and the gradient calculation and parameter update, efficient optimization of the large language model is achieved, the performance and computational efficiency of the model are improved, and the training cost is reduced.

[0095] For the method steps disclosed in the above embodiments, the method steps are described as a series of action combinations for the purpose of simple description, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0096] Any process or method description described in a flowchart or other manner may be understood as representing a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations, in which the order shown or discussed may not be followed, including executing and implementing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, or executing computer instructions and implementing the corresponding functions according to program structures such as loops and branches, which can be naturally understood by those skilled in the art when implementing the embodiments of the present invention.

[0097] like Figure 2 The automatic speech processing system based on the language large model is used to implement the automatic speech method based on the language large model, including:

[0098] The user voice data acquisition module is used to acquire user voice interaction data to form a multi-round dialogue data set, perform natural language processing on the dialogue data set using a large language model, and fine-tune the multi-round dialogue data set by annotation to generate training set data;

[0100] A language model fine-tuning module is used to fine-tune the language model using LoRA technology according to the training set data to form a fine-tuned large language model. The fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix into each level of the language model, adjusting the model parameters, and fine-tuning the preprocessed training set data;

[0101] The correction intention data generation module is used to perform semantic recognition on the multi-round dialogue data set based on the fine-tuned large language model to generate corresponding correction intention data.

[0102] The implementation methods of the system described above are merely illustrative, for example, the various functional modules, units or subsystems in the system may or may not be physically separated, or may or may not be physical units, that is, they may be located in the same place, or they may be distributed to multiple different systems and their subsystems or modules. Those skilled in the art may select some or all of the functional modules, units or subsystems according to actual needs to achieve the purpose of the embodiments of the present invention, and those of ordinary skill in the art may understand and implement the above-mentioned situations without creative work.

[0103] like Figure 3 and Figure 4 As shown in Figure 1, voice is an important interactive channel in the process of automobile intelligence. Facing different user requirements and statements, voice needs to understand user intentions more accurately, so enhancing semantic understanding capabilities is the key. Figure 3 In the case of a user's multi-round voice interaction, for the first "play George Lam's songs", the music intent was accurately identified and the singer was George Lam. However, for the second "how many scents does the fragrance system have", under normal circumstances, the voice should give the output intent and answer how many scents there are in total, but the voice did not recognize the user's semantics and treated it as a chat. As for why this semantics was not recognized, there may be many reasons. The most critical one is that the semantic model did not add samples of these statements during the training process, or there were only a few similar samples during training, which were not enough for the model to learn these semantics. Therefore, an important way to enhance semantic generalization and understanding is to enhance high-quality data samples. The question is how to mine high-quality data samples and add them to the model?

[0105] Under normal circumstances, it is necessary to find these cases and mark them manually, but the ideal state is to automate this process, not to fix the problem through bugs or user phone feedback. In fact, it is a good choice to use the ability of the big model. The big model has strong semantic understanding and generalization capabilities through learning from a lot of corpora. In particular, the current big model is being applied to various aspects and has obtained good results to verify its own capabilities and methods. Therefore, it is very easy to achieve full automation of semantic enhancement and generalization of semantic understanding based on the big model. This embodiment is based on the big model to automatically enhance the semantic understanding and generalization capabilities. First, the data of multiple rounds of user voice interactions are mined, and the interaction process between the user context and the voice is passed to the big model for evaluation. Based on the evaluation results of the big model, it is judged whether the user is satisfied with the results of this round of voice and whether there are problems with the voice semantics. After judging the semantic problems, the big model is asked to give key information such as the correct intention to form a data sample, and then the data is passed to the big model for generalization to generate multiple groups of data. The data is manually sampled regularly to check data quality problems. After being aggregated together, the sample data is stored in the database for semantic model training. After the model is trained, it is put back online, so that the entire process is automatically closed and continuously self-iterated and upgraded, so as to accurately identify user semantics to improve user experience, enhance user voice usage, and create an intelligent voice assistant that understands users better.

[0106] The process of the method for enhancing user semantic understanding and generalization based on a large model in this embodiment is as follows:

[0107] User voice session data mining, based on user behavior data in the data lake, aggregates each user and sorts them in chronological order. The next step is to divide it into each session. The session here refers to the interaction process between the user and the voice in a short period of time. For example, the user uses voice to operate the car control for 2 minutes after getting in the car, then plays music, and finally exits the voice. This is a session. We want to divide multiple rounds of data in a short period of time. In order to retain as much context as possible, it can get more context information when it is handed over to the large model, so as to make accurate judgments.

[0108] Furthermore, based on the large model, the user voice usage is evaluated:

[0109] ① User data processing: Aggregate by individual user and sort by time, and process the data into multiple rounds of conversations as follows:

[0110] [{

[0111] Sample: "Understand the following multi-round conversations with the car machine, determine whether the user is satisfied, point out which sentence is not satisfactory, and give the correct intention\n

[0112] User: Search for Wanxiang Fairy Tale World, Voice Reply: Although I don’t want to disappoint you, I don’t understand what you want to say

[0113] User: I mean I want to go to this place. Voice reply: OK, please tell me a place.

[0114] Tags: "Dissatisfied, Search for the fairy tale world of Vientiane, Navigation intent"

[0115] }

[0116] {

[0117] Sample: "Understand the following multi-round conversations with the car machine, determine whether the user is satisfied, point out which sentence is not satisfactory, and give the correct intention\n

[0118] User: How many scents does the fragrance system have? Voice reply: Let me learn and come back later

[0119] User: You are such a jerk that you don’t even know this. Voice reply: I will understand you better.”

[0120] Tags: "Dissatisfied, how many scents does the fragrance system have, system control intention"

[0121] }]

[0122] ② The above user data is processed into samples, and a batch of labels are manually annotated for fine-tuning the large model. After the large model is trained, the existing user data can be used to predict the model output label content.

[0123] The principle of fine-tuning the large model based on the lora method is as follows:

[0124] Lora is good for large model training. Models like GPT and Llama, which have billions of parameters, are usually too expensive to fine-tune for specific tasks or fields. LoRA retains the weights of the pre-trained model and adds trainable layers inside each model block. This results in a significant reduction in the number of parameters that need to be fine-tuned and significantly reduces GPU memory requirements.

[0125] This embodiment improves the accuracy of semantic understanding. It judges whether the semantics satisfy the user based on the actual user data based on the big model, greatly improves the judgment accuracy, and can accurately find the deficiencies of the semantic model and correct them using the big model. Secondly, the samples corrected by the big model are output to the semantic model for learning, so that the semantic model can learn these fundamentally and reduce the cost of the manual labeling process. This mode allows the semantic model to learn more and broader cases of negative feedback from real users, continuously improve the accuracy of semantic understanding, and achieve a virtuous circle.

[0127] This embodiment improves the generalization ability. The big model digs out the semantically problematic cases and corrects them. For the corrected cases, the big model is used to generate more similar cases. Finally, these generalized cases are used as samples for the semantic model to learn, so as to learn from one example and apply it to other cases.

[0128] This embodiment reduces costs and improves real-time performance. It uses a large model to identify real negative feedback cases from users and then lets the large model make corrections. This mode is much cheaper and faster than manual identification and labeling. At the same time, if this process can be run in real time within a short period of time, it can be achieved that the user’s statement that was not satisfied yesterday can be satisfied today when the new semantic model is launched. This real-time performance greatly improves the user’s experience of voice.

[0129] This embodiment realizes automation and a virtuous cycle. Based on data mining, large models, and semantic model training and online launch, this process can be automated and can solve user problems very accurately. When a new semantic model is obtained and new problems are discovered, the training and online launch cycle begins again, ultimately achieving a virtuous cycle.

[0130] like Figure 5 As shown, the present invention not only provides an automated speech processing method and system based on a large language model, but also provides a corresponding intelligent cockpit, electronic equipment and storage medium:

[0131] A smart cockpit is provided with the automatic speech processing system based on the language large model, and executes the automatic speech processing method based on the language large model.

[0132] An electronic device comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0133] A computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0134] Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 The electronic device provided in the embodiment of the present invention includes: one or more processors 710 and a storage device 720; the processor 710 in the electronic device may be one or more, Figure 5A processor 710 is taken as an example; the storage device 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement the automated speech processing method based on a large language model as described in any one of the embodiments of the present invention.

[0135] The electronic device may further include: an input device 730 and an output device 740 .

[0136] The processor 710, storage device 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0137] The storage device 720 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the automatic speech processing method based on the language large model provided in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the storage device 720, that is, the automatic speech processing method based on the language large model in the above method embodiment is implemented.

[0138] The storage device 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0141] Furthermore, when one or more programs included in the above-mentioned electronic device are executed by the one or more processors 710, the program executes the steps of the automated speech processing method based on the language large model.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the embodiments claimed in the claims can be used in any combination in the embodiments of the present invention.

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0145] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0146] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any way. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features. Throughout the specification, the same reference numerals indicate the same elements.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated speech processing method based on a large language model, applied to an automated speech processing system based on a large language model, It is characterized in that include: Obtain user voice interaction data to form a multi-round dialogue data set, use the language model to perform natural language processing on the dialogue data set, and fine-tune the multi-round dialogue data set through annotation to generate training set data; Fine-tune the language model using LoRA technology according to the training set data to form a fine-tuned large language model, wherein the fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix into each level of the language model, adjusting the model parameters, and fine-tuning the preprocessed training set data; Based on the fine-tuned large language model, semantic recognition is performed on the multi-round dialogue dataset to generate corresponding correction intention data.

2. The method for automatic speech processing based on a large language model according to claim 1, It is characterized in that The method further includes: obtaining user voice interaction data to form a multi-round dialogue data set, performing natural language processing on the dialogue data set using a large language model, and fine-tuning the multi-round dialogue data set by annotation to generate training set data. Acquire the voice interaction data between the user and the intelligent system, and process it through the context retention algorithm to form a multi-round dialogue data set. The context retention algorithm is specifically: sequence model and attention mechanism algorithm; Label the multi-round dialogue dataset, and perform natural language processing on the labeled multi-round dialogue dataset based on the language big model to obtain natural language understanding data for identifying user intent and evaluating dialogue satisfaction; Fine-tune the natural language understanding data, adjust the parameters of the large language model through the back-propagation algorithm and the gradient descent optimization method, calculate the gradient of the large language model parameters through the loss function, and use the Adam or SGD optimization algorithm to update the parameters, and perform iterative training to minimize the difference between the prediction results of the large language model and the actual annotations.

3. The method for automatic speech processing based on a large language model according to claim 1, It is characterized in that The method further comprises: fine-tuning the large language model using the LoRA technology according to the training set data to form a fine-tuned large language model. Determine the trainable low-rank matrix of each layer in the fine-tuned large language model, and use the data in the training set to calculate the gradient of the low-rank matrix through the back-propagation algorithm; According to the gradient update rule, the low-rank matrix is ​​updated to optimize the fit of the model to the training set; Vectorize the user voice data and convert it into numerical features suitable for model input.

4. The method for automatic speech processing based on a large language model according to claim 3, It is characterized in that The vectorization processing of the user voice data is specifically: performing denoising and normalization processing on the user voice data, and applying Fourier transform to convert the voice signal into frequency domain data; The principal component analysis is performed on the feature vector obtained after the vectorization process to reduce the dimension, and the representative feature vector is extracted, and the representative feature vector is standardized.

5. The method for automatic speech processing based on a large language model according to claim 1, It is characterized in that The method further comprises: fine-tuning the large language model using the LoRA technology according to the training set data to form a fine-tuned large language model. Modify the weight matrix of the large language model and adjust the model parameters by multiplying the weight matrix with the low-rank matrix; Decompose the parameter matrix of each level of the model, and use the singular value decomposition (SVD) algorithm to initialize the low-rank matrix of each level to capture the change direction of the parameter matrix; The back-propagation algorithm is used to calculate the gradient of each low-rank matrix relative to the loss function, and the loss function is derived by the chain rule to obtain the gradient information of each low-rank matrix element.

6. The method for automatic speech processing based on a large language model according to claim 5, It is characterized in that The modification of the weight matrix of the large language model is specifically as follows: Analyze the weight matrix in the large language model to determine the relationship between the weight matrix and the model performance; Construct a low-rank matrix that matches the dimension of the weight matrix, and multiply the weight matrix with the low-rank matrix through multiplication operation to generate an adjusted weight matrix; Using the training set data, the gradient of the adjusted weight matrix is ​​calculated through the back propagation algorithm, and the low-rank matrix is ​​updated using the optimization algorithm based on the calculated gradient.

7. An automated speech processing system based on a large language model, used to implement the automated speech method based on a large language model as claimed in any one of claims 1 to 6, It is characterized in that include: The user voice data acquisition module is used to acquire user voice interaction data to form a multi-round dialogue data set, perform natural language processing on the dialogue data set using a large language model, and fine-tune the multi-round dialogue data set by annotation to generate training set data; A language model fine-tuning module is used to fine-tune the language model using LoRA technology according to the training set data to form a fine-tuned large language model. The fine-tuned large language model specifically comprises: preprocessing the training set to generate preprocessed training set data, introducing a low-rank matrix into each level of the language model, adjusting the model parameters, and fine-tuning the preprocessed training set data; The correction intention data generation module is used to perform semantic recognition on the multi-round dialogue data set based on the fine-tuned large language model to generate corresponding correction intention data.

8. A smart cockpit, It is characterized in that The smart cockpit is provided with the automated speech processing system based on the language large model as claimed in claim 7, and executes the automated speech processing method based on the language large model as claimed in any one of claims 1 to 6.

9. An electronic device, It is characterized in that include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 6.

10. A computer-readable storage medium, It is characterized in that It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method described in any one of claims 1 to 6.

Citation Information

Cited By

  • Training method of dialect voice cloning model, dialect voice cloning method and device, terminal equipment and storage medium

    CN119943027A

  • A training method for a dialect voice cloning model, a dialect voice cloning method, a device, a terminal device, and a storage medium

    CN119943027B

  • Speech recognition and natural language processing integration method and system

    CN120220652A

  • Electronic medical record text normalization method based on large model fine tuning

    CN120690362A

  • Training data synthesis method of voice assistant model, voice assistant system and computer equipment

    CN120783729A