Telecommunication service question and answer method, system and server based on large model fine tuning

Abstract

The invention discloses a telecommunication service question and answer method and system based on large model fine tuning and a server. The method comprises the following steps: marking customer service historical chat record data to form a first data set; acquiring problem data in the internal telecommunication service knowledge base to form a second data set; after a third data set is formed by constructing a prompt text input text model for model data construction, the three data sets are integrated and cleaned into a training data set in a json data format, a low-rank matrix is introduced, distributed training is adopted to carry out fine tuning training on an initial large language model, and a trained large language model is obtained. And finally, performing question rewriting and vectorization on a to-be-answered text sent by a user, then calculating with vectors in the telecommunication service knowledge base, searching for a sentence with the highest similarity, performing correlation evaluation through the rearrangement model, and inputting a rewritten statement with the correlation lower than a set value into the trained large language model to obtain a corresponding reply text. The efficiency and the accuracy of telecommunication service questioning and answering are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a telecommunication business question-answering method, system and server based on large model fine-tuning. Background Art

[0002] The telecommunications knowledge question-answering system plays a core role in the telecommunications industry. Relying on the powerful contextual understanding and generation capabilities of large language models, it is committed to providing users with accurate and immediate telecommunications business consulting services. These large language models have mastered a wealth of language expression and semantic analysis skills thanks to the pre-training of large-scale text data. However, it is worth noting that although large language models have broad application potential, their high training costs limit their popularity, making it difficult for many companies to afford the resources and technical support required for training. More importantly, existing large language models often lack an understanding of the professionalism and complexity of specific fields, resulting in poor results in answering telecommunications business-related questions, and are prone to misleading answers or lack of credibility. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a telecommunication business question-answering method based on large model fine-tuning, comprising the following steps:

[0004] S1, marking the customer service chat record data and marking the user questions to form a first data set; collecting the question data in the internal telecommunications business knowledge base to form a second data set; constructing a prompt text input text model to construct model data to form a third data set, integrating the first data set, the second data set and the third data set and removing duplicate data, deleting missing data in the field and unifying the fields to form a training data set in json data format;

[0005] S2, by introducing a low-rank matrix and using distributed training to fine-tune the initial large language model, a trained large language model is obtained;

[0006] S3, after rewriting the question to be answered sent by the user, the rewritten sentence is vectorized and calculated with the vector in the telecommunications business knowledge base, and the sentence with the highest similarity is found as the initial search result. The relevance of the initial search result is evaluated through the rearrangement model, and the rewritten sentence with a relevance lower than the set value is input into the trained large language model to obtain the corresponding reply text.

[0007] Preferably, fine-tuning training is performed on the initial large language model, specifically including: fine-tuning training on the large language model Qwen2, where the model weights are configured as W ∈ Rd×d, where d are the dimensions of the input and output respectively; in the process of starting fine-tuning, by introducing two smaller matrices A ∈ Rd×r, matrix B ∈ Rr×d, where r << d; initializing A through a random Gaussian distribution and initializing B with a zero matrix, fixing the parameters of the pre-trained model during training and only training matrices A and B; that is, AB = 0, and W0 is the initialized W; in order to approximate full-scale fine-tuning, a new weight initialization method is adopted: W = (W0 - A0B0) + AB; adopting the LoRA fine-tuning method, where the update formulas for matrices A, B, and W are:

[0008] where L is the loss function and η is the learning rate; minimizing the value where is the square of the Frobenius norm of the matrix, When G0 is a non-negative diagonal matrix: A0 = (I n ) [:,:r] ; B0 = (I m ) [r:2r,:] ; where I n 、I m are the n-order and m-order identity matrices respectively, [:,:r] and [r:2r,:] take the first r columns and the r + 1 to 2r rows; when G0 is not a diagonal matrix:

[0009] where A0 = U(I n ) [:,:r] = U [:,:r] , B0 = (I m ) [r:2r,:] V = V [r:2r,:] ; The G0 matrix is SVD to UPV, U ∈ R n×n , V ∈ R m×m is an orthogonal matrix, and P ∈ R n×m is a diagonal matrix; by using the samples in the training dataset, calculate the initial gradient The G0 matrix is SVD to UPV, the first r columns of U initialize A, and the r + 1 to 2r rows of V initialize B, and the trained W1 is obtained.

[0010] Preferably, fine-tuning training on the initial large language model also includes: performing LoRA optimization based on the attention mechanism, and applying the attention weight matrices W q , W k , W v , W o, perform low-rank decomposition, where the attention weight is calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; w′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈R d×r , B v ∈R r×d , A o ∈R nd×r , B o ∈R r×nd .

[0011] Preferably, fine-tuning the initial large language model also includes: using a deep learning training optimization library to perform distributed training, performing data parallel training, optimizer parallel training, gradient parallel training and model parallel training.

[0012] Preferably, the initial large language model is fine-tuned and trained, specifically including: using N training processors to perform data parallel training operations, independently running an initial large language model in each training processor, and the model weights will be divided into N parts and each training processor will have one part; after each iteration, the Dataset data is divided into N parts and sent to each training processor respectively; before performing forward propagation calculation, the weights distributed on each training processor are retrieved by performing a full aggregation algorithm operation, and the weights are formed into complete parameters for forward calculation, and the weights that are not maintained by themselves are discarded after the calculation is completed; before performing reverse propagation calculation, the weights are retrieved by performing a full aggregation algorithm operation on the weights, and the weights distributed on each training processor are retrieved, and the complete parameters are formed for reverse propagation calculation, and the weights that are not maintained by themselves are discarded after the calculation is completed; after performing forward and reverse propagation calculations, each training processor obtains a gradient, and each training processor independently calculates according to the obtained data The gradient is calculated and the full reduction algorithm is called to calculate the gradient mean. Each training processor then updates the parameters independently. The gradients that are not maintained by the corresponding training processor after aggregation are removed. The corresponding weights are updated for each training processor using the maintained optimizer and gradients. Finally, the full aggregation algorithm is performed on the weights to aggregate the weights on other nodes. The full reduction algorithm is performed on the gradients to ensure that the gradients maintained by each training processor are aggregated gradients, and the gradients that are not maintained by the corresponding training processor after aggregation are removed. On the basis of data parallelism, each training processor accumulates part of the gradients on each node and broadcasts the results to all nodes, so that all nodes have a complete gradient. The full aggregation algorithm is performed on the weights to aggregate the weights on other nodes, and the updated weights are synchronized to each training processor and their respective optimizers. After the training is completed, the weights are merged, and the matrix A and matrix B parameters are added to the parameter W0 to complete the training of the entire large language model.

[0013] The present invention also discloses a telecommunications service Q&A system based on large model fine-tuning, including a data construction module, a fine-tuning training module, and a vector search module. The data construction module is used to label the customer service historical chat record data and annotate user questions to form a first data set; collect various question data in the internal telecommunications service knowledge base to form a second data set; construct a third data set through a prompt text input text model for model data construction, and integrate the first data set, the second data set, and the third data set, and after removing duplicate data, deleting data with missing fields, and unifying fields, integrate them into a training data set in json data format; the fine-tuning training module is used to fine-tune and train the initial large language model by introducing a low-rank matrix and adopting distributed training to obtain a trained large language model; the vector search module is used to rewrite the question of the text to be answered sent by the user, vectorize the rewritten statement, calculate it with the vectors in the telecommunications service knowledge base, find the sentence with the highest similarity as the initial retrieval result, evaluate the relevance of the initial retrieval result through a re-ranking model, and input the rewritten statement with a relevance lower than the set value into the trained large language model to obtain the corresponding reply text.

[0014] Preferably, the fine-tuning training module specifically includes: fine-tuning and training the large language model Qwen2, where the model weights are configured as W∈Rd×d, where d are the dimensions of the input and output respectively; in the process of starting fine-tuning, by introducing two smaller matrices A∈Rd×r, matrix B∈Rr×d, where r<<d; initializing A through a random Gaussian distribution and initializing B with a zero matrix, fixing the parameters of the pre-trained model during training, and only training matrices A and B; that is, AB = 0, and W0 is the initialized W; in order to approximate full-scale fine-tuning, a new weight initialization method is adopted: W=(W0 - A0B0)+AB; adopting the Lora fine-tuning method, where the update formulas for matrices A, B, and W are:

[0015] where L is the loss function and η is the learning rate; minimize the value where is the square of the Frobenius norm of the matrix, When G0 is a non-negative diagonal matrix: A0=(I n ) [:,:r] ; B0=(I m ) [r:2r,:] ; where I n and I m are the n-order and m-order identity matrices respectively, [:,:r] and [r:2r,:] take the first r columns and the r + 1 to 2r rows; when G0 is not a diagonal matrix:

[0016] Where A0=U(I n ) [:,:r] =U [:,:r] , B0=(I m ) [r:2r,:] V=V [r:2r,:] ; The SVD of the G0 matrix is ​​UPV, U∈R n×n , V∈R m×m is an orthogonal matrix, P∈R n×m is a diagonal matrix; the initial gradient is calculated by using the samples in the training data set The matrix SVD is UPV, the first r columns of U initialize A, the r+1~2r rows of V initialize B, and the trained W1 is obtained.

[0017] Preferably, the fine-tuning training module hiatus includes: performing LoRA optimization based on the attention mechanism, converting the attention weight matrix W q ,W k ,W v ,W o , perform low-rank decomposition, where the attention weight is calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; v′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈R d×r , B v ∈R r×d , A o ∈Rnd ×r , B o ∈R r×nd .

[0018] The present invention also discloses a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods when executing the computer program.

[0019] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the aforementioned methods are implemented.

[0020] The present invention discloses a telecommunication business question-and-answer method, system and server based on large model fine-tuning, which obtains training data through three methods: marking customer service historical chat record data, collecting question data in an internal telecommunication business knowledge base, and constructing prompt text data to construct model data, and then constructs a training data set in the telecommunication field after cleaning and integration, and then fine-tunes the initial large language model by introducing a low-rank matrix and adopting a distributed training method, so that under the condition of limited training resources, the model can fully understand and adapt to the special needs of the telecommunication field, obtain the desired answer according to the input user text question, rewrite and vectorize the question sent by the user, and input the question with low similarity to the question in the existing knowledge base into the large model to obtain a new answer, thereby improving the accuracy and efficiency of the telecommunication question-and-answer system.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 A schematic diagram of the steps of a telecommunication business question-answering method based on large model fine-tuning disclosed in one embodiment of the present invention.

[0024] Figure 2 A schematic diagram of the structure of the distributed training of the model disclosed in one embodiment of the present invention.

[0025] Figure 3 A schematic diagram of the structure of a vectorized search disclosed in an embodiment of the present invention.

[0026] Figure 4It is a schematic diagram of the structure of a telecommunication business question-answering system based on large model fine-tuning disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0030] Unless otherwise defined, the technical or scientific terms used herein shall have the common meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but indicate the existence of at least one.

[0031] The telecommunications knowledge question-and-answer system plays a core role in the telecommunications industry. Relying on the powerful contextual understanding and generation capabilities of large language models, it is committed to providing users with accurate and immediate telecommunications business consulting services. . These large language models have benefited from the pre-training of large-scale text data and have mastered a wealth of language expression and semantic analysis skills. However, it is worth noting that although large language models have broad application potential, their high training costs limit their popularity, making it difficult for many companies to afford the resources and technical support required for training. More importantly, existing large language models often lack an understanding of the professionalism and complexity of specific fields, resulting in poor results in answering telecommunications business-related questions, and are prone to misleading answers or lack of credibility. Therefore, in this embodiment, a telecommunications business question-and-answer method based on large model fine-tuning is disclosed, as shown in the attached figure. Figure 1 As shown, the following steps are included.

[0032] Step S1, label the customer service historical chat record data and mark user questions to form a first data set; collect the data of each question in the internal telecommunications business knowledge base to form a second data set; construct a prompt text input text model to construct model data to form a third data set, integrate the first data set, the second data set and the third data set and remove duplicate data, delete fields with missing data and unify fields to form a training data set in json data format.

[0033] Specifically, the constructed training data set includes two types of data: single-round conversation data and multi-round conversation data, which mainly come from customer service chat records, internal and external knowledge bases, and model construction. Specifically, customer service chat records are information generated by telecom personnel when interacting with users, which contain data such as user questions, customer service answers, and related interaction information. Telecom personnel manually annotate user questions in customer service historical chat records, identify information such as the category, intention, or emotion of the questions, and construct a first data set as the basis for the training data set, which is used for subsequent model training and automatic classification and intention recognition of user questions. At the same time, the user questions are similarly expanded using the information in the telecom knowledge base combined with the model to construct a second data set, and new user question data is generated by constructing a third data set through a prompt text input text generation model, further expanding the training data set, ensuring the comprehensiveness of the training data set and further improving the accuracy and relevance of the model answer. In this embodiment, the text generation model used to construct new user questions is a commercial DeepSeek model.

[0034] In addition, since there is dirty data in the constructed data, in this embodiment, after the construction is completed, the data is integrated and cleaned to improve the data quality of the training dataset, including multiple steps such as removing duplicate data, deleting data with missing important fields, standardizing formats, unifying fields, and manual verification. Finally, the training dataset is synthesized and the data in it is saved in the json data format.

[0035] Step S2, fine-tune the initial large language model by introducing low-rank matrices and using distributed training to obtain a trained large language model.

[0036] Specifically, after the training data is constructed, it is input into the initial large language model, and combined with the LoRA fine-tuning technology, the initial large language model is fine-tuned by introducing two low-rank matrices A and B. Among them, the LoRA fine-tuning technology is an efficient fine-tuning method for large language models. Its core idea is to simulate the effect of full-parameter fine-tuning by introducing two low-rank matrices A and B to replace the increment of the original weight matrix during the model training process while keeping the original parameters of the initial large language model fixed, so as to reduce the number of parameters required for training and reduce the consumption of computing resources, that is, W = W0 + ΔW = W0 + AB, where W ∈ R d×d represents the weight matrix of the large language model parameters, W0 represents the weight matrix of the initial large language model, ΔW represents the increment of the large language model weight matrix during the fine-tuning training, A ∈ R d×r , B ∈ R r×d are the introduced low-rank matrices, d represents the dimension of the input and output, and r << d. In this embodiment, the large language model Qwen2 is used as the initial large language model, and this model supports long text input of 128K tokens. Specifically, the LoRA fine-tuning process specifically includes the following content.

[0037] Step S101, configure the weights of the large language model Qwen2 as W ∈ R d×d , where d are the dimensions of the input and output respectively.

[0038] Step S102, introduce two smaller matrices A ∈ Rd×r, matrix B ∈ Rr×d, and initialize matrices A and B so that W = (W0 - A0B0) + AB to obtain the initial values A0 and B0 of matrices A and B, where r << d, and W0 is the weight of the large language model Qwen2 in the initial state.

[0039] Specifically, the original LoRA fine-tuning initializes matrix A through a random Gaussian distribution and initializes matrix B with a zero matrix. In this embodiment, to make the LoRA fine-tuning approximate full-scale fine-tuning, a new weight initialization method is adopted to ensure that the initialized weight W0 = W. Among them, since this embodiment selects stochastic gradient descent as the optimizer, its full-scale fine-tuning gradient update formula is: At this time, the update formulas of matrices A, B and W are configured as:

[0040] To ensure that W1 after Lora fine-tuning is as close as possible to W1 after full fine-tuning, To minimize:

[0041]

[0042] in, is the square of the Frobenius norm of the matrix,

[0043] Specifically, when G0 is a non-negative diagonal matrix, we select specific A0 and B0 to simplify the problem, and take A0 = (I n ) [:,:r] ; B0=(I m ) [r:2r,:] , where I n ,I m They are n-order and m-order unit matrices respectively, [:,:r] and [r:2r,:] take the first r columns and r+1~2r rows.

[0044] When G0 is not a diagonal matrix, the singular value decomposition (SVD) of G0 is used to find the appropriate representation of A0 and B0:

[0045]

[0046] Where A0=U(I n ) [:,:r] =U [:,:r] , B0=(I m ) [r:2r,:] V=V [r:2r,:] , G0=UPV,U∈R n×n , V∈R m×m is an orthogonal matrix, P∈R n×m is a diagonal matrix.

[0047] Then step S102 specifically also includes the following content.

[0048] Step S1021, calculate the initial gradient And perform singular value decomposition on the matrix G0 to obtain its corresponding matrices U, P, V, where L is the loss function, W0 is the weight of the large language model Qwen2 in the initial state, and U∈R n×n , V∈R m×m is an orthogonal matrix, P∈R n×m is a diagonal matrix.

[0049] Step S1022: Use the first r columns of matrix U to initialize matrix A to obtain A0, and use the r+1th row to the 2rth row of matrix V to initialize matrix B to obtain B0.

[0050] Step S103, fine-tune the large language model Qwen2 using the samples in the training data set to obtain the weight matrix W1 of the trained large language model Qwen2, where Where L is the loss function and η is the learning rate.

[0051] Furthermore, since the questions asked by users in the telecommunications intelligent question-answering system are generally long, the selected initial large language model usually includes an attention mechanism to ensure that the model has the ability to process complex contexts with long-range dependencies. Therefore, during the training of the initial large language model, LoRA optimization based on the attention mechanism can be performed to ensure the model's ability to process complex contexts and reduce the model's computational overhead. Then fine-tuning the initial large language model also includes: adjusting the attention weight matrix W q ,W k ,W v ,W o , perform low-rank decomposition, where the attention weight is calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; w′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈Rd×r , B v ∈R r×d , A o ∈R nd×r , B o ∈R r×nd .

[0052] Specifically, by substituting the weight matrix after low-rank decomposition into the formula of the self-attention mechanism, the attention weight matrix W q ,W k ,W v ,W o , performing low-rank decomposition, we can obtain the attention calculation method optimized by Lora, which greatly reduces the computing and storage requirements of the model without significantly affecting the performance.

[0053] In this embodiment, in order to further improve the training speed, distributed training can also be used for training, that is, fine-tuning the initial large language model also includes: using a deep learning training optimization library for distributed training, performing data parallel training, optimizer parallel training, gradient parallel training and model parallel training.

[0054] Specifically, this embodiment adopts the Deepspeed distributed training framework and turns on the ZeRO-3 optimizer to achieve data parallel training, optimizer parallel training, gradient parallel training and model parallel training. Since each computing node (such as a server or graphics card) in the distributed training environment holds part of the data or model parameters, the model training speed can be improved. Among them, DeepSpeed ​​is a deep learning acceleration framework developed by Facebook AI Research. It can run on a single GPU or perform distributed training on multiple GPUs and multiple machines, which can quickly reduce training costs and time. ZeRO is an optimizer designed to solve the memory bottleneck problem in large-scale distributed training. It is an optimizer designed to solve the memory bottleneck problem in large-scale distributed training.

[0055] Specifically, the data parallel training includes: using N training processors to perform data parallel training operations, independently running an initial large language model in each training processor, and the model weights will be divided into N parts and each training processor will have one part; after each iteration, the Dataset data will be divided into N parts and sent to each training processor respectively.

[0056] The parallel training of the model includes: performing a full aggregation algorithm operation on the weights to retrieve the weights distributed on each training processor before performing forward propagation calculations, forming complete parameters for forward propagation calculations, and discarding weights that are not maintained by itself after the calculations are completed; performing a full aggregation algorithm operation on the weights to retrieve the weights distributed on each training processor before performing reverse propagation calculations, forming complete parameters for reverse propagation calculations, and discarding weights that are not maintained by itself after the calculations are completed.

[0057] The parallel training of the optimizer specifically includes: after executing forward and back propagation calculations, each training processor obtains a gradient, each training processor independently calculates the gradient based on the obtained data and calls the full reduction algorithm to calculate the gradient mean, and each training processor then independently updates the parameters; removes the gradients that are not maintained by the corresponding training processor after aggregation, updates the corresponding weights for each training processor using the maintained optimizer and gradients, and finally performs a full aggregation algorithm operation on the weights to aggregate the weights on other nodes.

[0058] The gradient parallel training specifically includes: performing a full reduction algorithm operation on the gradient to ensure that the gradient maintained by each training processor is an aggregated gradient, and removing the gradient that is not maintained by the corresponding training processor after aggregation; on the basis of data parallelism, each training processor accumulates the partial gradients on each node and broadcasts the results to all nodes, so that all nodes have a complete gradient, performs a full aggregation algorithm operation on the weights to aggregate the weights on other nodes, and synchronizes the updated weights to each training processor and each optimizer.

[0059] In this embodiment, distributed training of a large language model mainly includes the following contents.

[0060] Step S201, evenly divide the data in the training data set into N training data blocks of the same size, and send them to N training processors that independently run the same initial large language model and respectively contain 1 / N weights of the initial large language model.

[0061] Specifically, the training environment is first initialized to provide a stable and reliable foundation for subsequent model training, and the training data set and model weights are evenly divided so that each GPU can independently perform the propagation process on its own data. Figure 2As shown in the figure, when N is equal to 2, the data in the training data set is evenly cut into two parts and sent to two identical GPU cards loaded with the initial large language model. In this embodiment, 7 A-800 graphics cards are selected for training, and the batch-size of each graphics card is set to 8, the initial learning rate is 5e-5, and the weights to be fine-tuned are specified as q_proj, k_proj, v_proj, and o_proj. During the entire training process, only these specified weights will be updated.

[0062] In step S202, each training processor uses the full gathering algorithm to retrieve the parameters of the weight composition distributed on each GPU, and performs a forward propagation calculation and a back propagation calculation according to the received training data block to obtain an independent gradient.

[0063] Specifically, after the configuration is completed, each GPU performs forward propagation and backward propagation in turn. However, before forward propagation and backward propagation, the full aggregation algorithm AllReduce is required to retrieve the parameters composed of the weights distributed on each GPU to ensure model consistency. After the parameter combination is completed, each CPU independently calculates a gradient based on the data obtained by each GPU card and the complete model parameters during the forward propagation and backward propagation process. Among them, the full aggregation algorithm is a commonly used data synchronization method in distributed training, which can accumulate the data (gradient or weight) of each node and broadcast it to all nodes.

[0064] Step S203, using the full aggregation algorithm to aggregate the gradients of each training processor, and calculate the average value of the overall gradient, and each training processor updates its own weight according to the gradient average value.

[0065] Specifically, using the full aggregation algorithm, the gradient calculated by each GPU card is sent to other GPUs for aggregation and the gradient average of all GPU cards is calculated, that is, each GPU will obtain the same gradient after aggregation. Each GPU uses the obtained gradient average to update and maintain the local model parameters. Each parameter is aggregated again before forward propagation and back propagation in step S202.

[0066] Step S204, repeat the above steps until the preset training stop condition is reached, merge the weights after the training is completed, add the matrix A and matrix B parameters to the parameter W0, and complete the training of the entire large language model.

[0067] In the distributed training mode, the training process of a large language model is divided into multiple subtasks, which significantly speeds up the training time of the model.

[0068] Step S3, after rewriting the text to be answered sent by the user, the rewritten sentence is vectorized and calculated with the vector in the telecommunications business knowledge base, and the sentence with the highest similarity is found as the initial search result. The relevance of the initial search result is evaluated through the rearrangement model, and the rewritten sentence with a relevance lower than the set value is input into the trained large language model to obtain the corresponding reply text.

[0069] Specifically, after the model training is completed, the corresponding answer information can be obtained for the question statement input by the user, helping to improve the accuracy of the telecommunications business question and answer system. At the same time, in order to further improve the accuracy and efficiency of the telecommunications business question and answer system, this embodiment also combines the construction of a telecommunications business knowledge base and vector search to obtain the corresponding answers to user questions. Among them, the telecommunications business knowledge base is constructed based on telecommunications business question and answer knowledge, including two levels: provincial telecommunications business knowledge base and group telecommunications business knowledge base. When the platform configures new information for frequently asked questions (FAQ), the data in the provincial telecommunications business knowledge base and the group telecommunications business knowledge base can be automatically updated to ensure the real-time and accuracy of the data. As shown in the attached Figure 3 As shown, telecommunication knowledge will be input into the BGE model and converted into text vectors, and stored in the Milvus database that supports vector retrieval to form a telecommunication business knowledge base. When the user enters question information, the user question information is first rewritten to convert the question into a standardized format or terminology so as to better match the preset knowledge base or question-answering system. After rewriting, the rewritten sentence is input into the BGE model and converted into a text vector and vectorized with the data in the telecommunication business knowledge base. The sentences with the highest similarity to the user's question in the telecommunication business knowledge base are found and input into the Re_rank model for a deeper correlation evaluation and sorting, ensuring that the answer finally displayed to the user meets the user's query intention. When the sentence with the highest similarity to the user in the knowledge base is also lower than the preset similarity value, the user sentence is input into the trained large language model to generate a new answer. In the above manner, while ensuring the efficiency of the business system answer, the accuracy of the answer is improved.

[0070] The final response to user input mainly includes three types: single-round or multi-round FAQ answers, platform SOP calls, and model context answers.

[0071] FAQ single or multi-round answering: When the user's question falls within the scope of the standard knowledge base, we will directly retrieve and reply to the corresponding answer from the knowledge base. If the question is not in the knowledge base, we will use the large language model to analyze and answer it independently.

[0072] Platform SOP call: When we identify that a user's question belongs to a specific business type (such as traffic package recommendation, etc.), we will forward the answer through the platform's standard operating procedure (SOP) to ensure that the user receives accurate and professional answers.

[0073] Model contextual answer: When we recognize that a user's question involves contextual information, we will use the large response model and combine it with the previous conversation content to provide an answer that is more in line with the user's needs.

[0074] The present embodiment discloses a telecommunication business question-answering method based on large model fine-tuning, which obtains training data through three methods: marking customer service historical chat record data, collecting question data in an internal telecommunication business knowledge base, and constructing prompt text data to construct model data. After cleaning and integration, a training data set in the telecommunication field is constructed, and then a low-rank matrix is ​​introduced and the initial large language model is fine-tuned and trained by distributed training. In this way, under the condition of limited training resources, the model can fully understand and adapt to the special needs of the telecommunication field, obtain the desired answer based on the input user text question, rewrite and vectorize the question sent by the user, and input the question with low similarity to the question in the existing knowledge base into the large model to obtain a new answer, thereby improving the accuracy and efficiency of the telecommunication question-answering system.

[0075] In another embodiment, as shown in the attached Figure 4 As shown, a telecommunications business question-answering system based on large model fine-tuning is also disclosed, including a data construction module 1, a fine-tuning training module 2 and a vector search module 3. The data construction module 1 is used to mark the customer service historical chat record data and mark the user questions to form a first data set; collect the data of each question in the internal telecommunications business knowledge base to form a second data set; construct the model data by constructing a prompt text input text model to form a third data set, and integrate the first data set, the second data set and the third data set and remove duplicate data, delete the missing data of the field and unify the fields to form a training data set in json data format. The fine-tuning training module 2 is used to fine-tune the initial large language model by introducing a low-rank matrix and using distributed training to obtain a trained large language model. The vector search module 3 is used to rewrite the text to be answered sent by the user, vectorize the rewritten sentence, and calculate it with the vector in the telecommunications business knowledge base, find the sentence with the highest similarity as the initial search result, evaluate the relevance of the initial search result through the rearrangement model, and input the rewritten sentence with a relevance lower than the set value into the trained large language model to obtain the corresponding reply text.

[0076] In this embodiment, the fine-tuning training module specifically includes: fine-tuning and training the large language model Qwen2, where the model weights are configured as W ∈ Rd×d, where d is the dimension of the input and output respectively; during the start of the fine-tuning process, by introducing two smaller matrices A ∈ Rd×r and matrix B ∈ Rr×d, where r << d; initializing A through a random Gaussian distribution and initializing B with a zero matrix, fixing the parameters of the pre-trained model during training, and only training matrices A and B; that is, AB = 0, and W0 is the initialized W; where in order to approximate full-scale fine-tuning, a new weight initialization method is adopted: W = (W0 - A0B0) + AB; adopting the LoRA fine-tuning method, where the update formulas for matrices A, B, and W are:

[0077] where L is the loss function and η is the learning rate; minimizing the value where is the square of the Frobenius norm of the matrix, When G0 is a non-negative diagonal matrix: A0 = (I n ) [:,:r] ; B0 = (I m ) [r:2r,:] ; where I n and I m are the n-order and m-order identity matrices respectively, [:,:r] and [r:2r,:] take the first r columns and the r + 1 to 2r rows; when G0 is not a diagonal matrix:

[0078] where A0 = U(I n ) [:,:r] = U [:,:r] , B0 = (I m ) [r:2r,:] V = V [r:2r,:] ; The G0 matrix is SVD to UPV, U ∈ R n×n , V ∈ R m×m is an orthogonal matrix, and P ∈ R n×m is a diagonal matrix; by using the samples in the training dataset, calculate the initial gradient The G0 matrix is SVD to UPV, the first r columns of U initialize A, and the r + 1 to 2r rows of V initialize B, and the trained W1 is obtained.

[0079] In this embodiment, the fine-tuning training module hiatus includes: performing LoRA optimization based on the attention mechanism, and performing low-rank decomposition on the attention weight matrices W q , W k , W v , W o , where the attention weights are calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; w′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈R d×r , B v ∈R r×d , A o ∈R nd×r , B o ∈R r×nd .

[0080] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and similar parts between the various embodiments can be referred to each other. For the telecommunications business question-and-answer system based on large model fine-tuning disclosed in the embodiment, since it corresponds to the telecommunications business question-and-answer method based on large model fine-tuning disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the aforementioned method part.

[0081] In some other embodiments, a server is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the various steps of the telecommunication business question-answering method based on large model fine-tuning as described in the above embodiments when executing the computer program. The server may include, but is not limited to, a processor and a memory. It can be understood by those skilled in the art that the schematic diagram is only an example of a server and does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0082] If the server is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned embodiments of the telecommunications business question-and-answer method based on large model fine-tuning can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0083] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

Claims

1. A telecommunication business question answering method based on large model fine-tuning, characterized in that: It includes the following steps: S1, label the customer service historical chat record data and label user questions to form a first data set; collect each question data in the internal telecommunications business knowledge base to form a second data set; Construct a third data set by constructing a prompt text input text model for model data construction, and integrate the first data set, the second data set and the third data set, remove duplicate data, delete data with missing fields, and unify fields, and then integrate them into a training data set in json data format; S2, fine-tune and train the initial large language model by introducing a low-rank matrix and adopting distributed training to obtain a trained large language model; S3, rewrite the question of the text to be answered sent by the user, vectorize the rewritten statement, calculate it with the vectors in the telecommunications business knowledge base, find the sentence with the highest similarity as the initial retrieval result, evaluate the relevance of the initial retrieval result through a re-ranking model, and input the rewritten statement with a relevance lower than the set value into the trained large language model to obtain the corresponding reply text.

2. The telecommunication business question-answering method based on large model fine-tuning according to claim 1, characterized in that: The fine-tuning training of the initial large language model specifically includes: Fine-tune and train the large language model Qwen2, where the model weights are configured as W∈Rd×d, where d are the dimensions of the input and output respectively; In the process of starting fine-tuning, introduce two smaller matrices A∈Rd×r, matrix B∈Rr×d, where r<<d; initialize A through a random Gaussian distribution, initialize B with a zero matrix, and fix the parameters of the pre-trained model during training, only train matrices A and B; that is, AB = 0, and W0 is the initialized W; where a new weight initialization method is adopted to approximate full-scale fine-tuning: W=(W0 - A0B0)+AB; The Lora fine-tuning method is adopted, where the update formulas of matrix A, matrix B and W are: Where L is the loss function and η is the learning rate; Minimize the value in is the square of the Frobenius norm of the matrix, When G0 is a non-negative diagonal matrix: A0=(I n ) [:,:r] ; B0=(I m ) [r:2r,:] ; where I n ,I m They are n-order and m-order unit matrices respectively, [:,:r] and [r:2r,:] take the first r columns and r+1~2r rows; when G0 is not a diagonal matrix: Where A0=U(I n ) [:,:r] =U [:,:r] , B0=(I m ) [r:2r,:] V=V [r:2r,:] ; The SVD of the G0 matrix is ​​UPV, U∈R n×n , V∈R m×m is an orthogonal matrix, P∈R n×m is a diagonal matrix; By using the samples in the training dataset, the initial gradient is calculated The SVD of the G0 matrix is ​​UPV, the first r columns of U initialize A, the r+1~2r rows of V initialize B, and the trained W1 is obtained.

3. The telecommunication business question-answering method based on large model fine-tuning according to claim 2, characterized in that: The fine-tuning training of the initial large language model also includes: Perform LoRA optimization based on the attention mechanism and convert the attention weight matrix W q ,W k ,W v ,W o , perform low-rank decomposition, where the attention weight is calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; w′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈R d×r , B v ∈R r×d , A o ∈R nd×r , B o ∈R r×nd .

4. The telecommunication business question-answering method based on large model fine-tuning according to claim 3, characterized in that: The fine-tuning training of the initial large language model also includes: adopt a deep learning training optimization library for distributed training, and perform data parallel training, optimizer parallel training, gradient parallel training and model parallel training.

5. The telecommunication business question-answering method based on large model fine-tuning according to claim 4, characterized in that: The fine-tuning training of the initial large language model specifically includes: Use N training processors for data parallel training operations. An initial large language model runs independently in each training processor. The model weights will be divided into N parts and each training processor has one part; after each iteration, the Dataset data will be divided into N parts and sent to each training processor respectively; Before forward propagation calculation, perform an all-gather algorithm operation on the weights to retrieve the weights distributed on each training processor, form a complete parameter for forward calculation, and discard the weights that do not belong to its own maintenance after the calculation; Before backward propagation calculation, perform an all-gather algorithm operation on the weights to retrieve the weights distributed on each training processor, form a complete parameter for backward propagation calculation, and discard the weights that do not belong to its own maintenance after the calculation; After performing forward and backward propagation calculations, each training processor obtains a copy of the gradient. Each training processor independently calculates the gradient based on the data obtained and calls the all-reduce algorithm to calculate the mean gradient. Each training processor then independently updates the parameters; remove the gradients that are not maintained by the corresponding training processor after summarization, update the corresponding weights for each training processor using the maintained optimizer and gradient, and finally perform an all-gather algorithm operation on the weights to summarize the weights on other nodes; Perform an all-reduce algorithm operation on the gradients to ensure that the gradients maintained by each training processor are aggregated gradients, and remove the gradients that are not maintained by the corresponding training processor after summarization; on the basis of data parallelism, each training processor accumulates the partial gradients on each node and broadcasts the result to all nodes, so that all nodes have a copy of the complete gradient, perform an all-gather algorithm operation on the weights to summarize the weights on other nodes, and synchronize the updated weights to each training processor and their respective optimizers; After training is completed, merge the weights, add the parameters of matrix A and matrix B to parameter W0 to complete the training of the entire large language model.

6. A telecommunications business question-answering system based on large model fine-tuning, characterized in that: Including: A data construction module for labeling the customer service historical chat record data and annotating user questions to form a first data set; Collect various question data in the internal telecommunications business knowledge base to form a second data set; Construct a third data set through a prompt text input text model for model data construction, and integrate the first data set, the second data set and the third data set, remove duplicate data, delete data with missing fields, and unify the fields to form a training data set in json data format; A fine-tuning training module for fine-tuning the initial large language model by introducing low-rank matrices and using distributed training to obtain a trained large language model; A vector search module for rewriting the question of the text to be answered sent by the user, vectorizing the rewritten statement, calculating it with the vectors in the telecommunications business knowledge base, finding the sentence with the highest similarity as the initial retrieval result, evaluating the relevance of the initial retrieval result through a re-ranking model, and inputting the rewritten statement with a relevance lower than the set value into the trained large language model to obtain the corresponding reply text.

7. The telecommunication business question-answering system based on large model fine-tuning according to claim 6, characterized in that: The fine-tuning training module specifically includes: Fine-tune the large language model Qwen2, where the model weights are configured as W ∈ Rd×d, where d are the dimensions of the input and output respectively; During the start of the fine-tuning process, introduce two smaller matrices A ∈ Rd×r, matrix B ∈ Rr×d, where r << d; initialize A with a random Gaussian distribution and initialize B with a zero matrix. During training, fix the parameters of the pre-trained model and only train matrices A and B; that is, AB = 0, and W0 is the initialized W; in order to approximate full-scale fine-tuning, a new weight initialization method is adopted: W = (W0 - A0B0) + AB; The Lora fine-tuning method is adopted, where the update formulas of matrix A, matrix B and W are: Where L is the loss function and η is the learning rate; Minimize the value in is the square of the Frobenius norm of the matrix, When G0 is a non-negative diagonal matrix: A0=(I n ) [:,:r] ; B0=(I m ) [r:2r,:] ; where I n ,I m They are n-order and m-order unit matrices respectively, [:,:r] and [r:2r,:] take the first r columns and r+1~2r rows; when G0 is not a diagonal matrix: Where A0=U(I n ) [:,:r] =U [:,:r] , B0=(I m ) [r:2r,:] V=V [r:2r,:] ; The SVD of the G0 matrix is ​​UPV, U∈R n×n , V∈R m×m is an orthogonal matrix, P∈R n×m is a diagonal matrix; By using the samples in the training dataset, the initial gradient is calculated The SVD of the G0 matrix is ​​UPV, the first r columns of U initialize A, the r+1~2r rows of V initialize B, and the trained W1 is obtained.

8. The telecommunication business question-answering system based on large model fine-tuning according to claim 7, characterized in that: The fine-tuning training module hiatus includes: performing LoRA optimization based on the attention mechanism, and converting the attention weight matrix W q ,W k ,W v ,W o , perform low-rank decomposition, where the attention weight is calculated as: Where Q = XW q , K = XW k , V = XW v , W O is the linear transformation matrix, To W q , W k , W v , W O Perform low-rank decomposition: w′ q =W q +A q B q ; w′ k =W k +A k B k ; w′ v =W v +A v B v ; w′ o =W o +A o B o ; Among them A q ∈R d×r , A k ∈R d×r , A v ∈R d×r , B v ∈R r×d , A o ∈R nd×r , B o ∈R r×nd .

9. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1-7.

Images