Intelligent question answering methods, electronic devices and storage media
By matching user questions in an industrial knowledge base and generating answers using an industrial large language model, the problem of low question-and-answer efficiency caused by the complexity of the industrial knowledge system is solved, and an efficient and accurate question-and-answer service is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FU TAI HUA IND SHENZHEN
- Filing Date
- 2025-01-22
- Publication Date
- 2026-07-31
AI Technical Summary
The industrial knowledge system is vast and complex, making it difficult for users to quickly and accurately obtain the answers they need through question-and-answer methods.
By acquiring the question text input by the user, and utilizing a pre-set question base and industrial knowledge base, combined with an industrial large language model, the answer text is generated through reasoning, thereby improving the efficiency of question answering.
It significantly improves question-and-answer efficiency, reduces labor costs, and enhances the accuracy and practicality of answers, providing support for the digital transformation of the industrial sector.
Smart Images
Figure CN122491455A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an intelligent question-answering method, electronic device, and storage medium. Background Technology
[0002] In the industrial production sector, users often encounter various complex problems during actual operation and decision-making. These problems drive them to frequently consult or search for specialized industrial knowledge in order to find effective solutions. However, the industrial knowledge system is not only vast and intricate, but its content is also often stored in diverse and fragmented ways. Due to the dispersed nature and heterogeneous formats of knowledge, coupled with the lack of a unified management and integration mechanism, users often find it difficult to quickly and accurately obtain the answers they need through question-and-answer methods. Summary of the Invention
[0003] In view of the above, it is necessary to provide an intelligent question-answering method, electronic device, and storage medium that can solve the technical problem of poor efficiency in intelligent question answering.
[0004] On the one hand, this application provides an intelligent question answering method, the method comprising: acquiring a question text input by a user; determining a matching target question from a preset question library based on the question text; retrieving industrial knowledge matching the question text from a preset industrial knowledge library according to the category of the target question; generating a prompt context based on the industrial knowledge and the question text; and using an industrial large language model to reason about the prompt context to generate an answer text corresponding to the question text.
[0005] In some embodiments of this application, the question library includes multiple preset questions, and each preset question corresponds to a vector representation. The step of determining the matching target question from the preset question library based on the question text includes: vectorizing the question text to obtain a question vector, and determining the target question from the multiple preset questions based on the matching of the question vector with the multiple vector representations.
[0006] In some embodiments of this application, determining the target problem from the plurality of preset problems based on matching the problem vector with the vector representation includes: determining a plurality of clusters based on clustering the plurality of vector representations, calculating the distance between the problem vector and the centroid of each cluster, selecting candidate clusters from the plurality of clusters based on the distance, and determining the target problem from the preset problems corresponding to the vector representations in the candidate clusters based on the similarity between the problem vector and the vector representations in the candidate clusters.
[0007] In some embodiments of this application, the training method of the industrial large language model includes: acquiring industrial corpus as training samples, wherein the industrial corpus includes one or more of equipment identifiers, technical terms and exception codes; and training the base large language model using a Bayesian algorithm and a low-rank adaptive fine-tuning LoRA method based on the training samples to obtain the industrial large language model.
[0008] In some embodiments of this application, the training samples include a first training sample and a second training sample, wherein the sample size of the first training sample is less than that of the second training sample. The step of training the large-base language model using the Bayesian algorithm and the low-rank adaptive fine-tuning LoRA method to obtain the industrial large-scale language model includes: based on the large-base language model and the first training sample, using the Bayesian algorithm to determine the expected rank value, based on the preset matrix and the expected rank value to determine the target low-rank matrix, and based on the second training sample and the target low-rank matrix, using the LoRA method to train the large-base language model to obtain the industrial large-scale language model.
[0009] In some embodiments of this application, determining the expected rank value based on the large-base language model and the first training sample using the Bayesian algorithm includes: constructing a prior distribution of multiple variables using the Bayesian algorithm; adjusting the large-base language model based on the first training sample and the prior distribution; and inputting the first training sample into the large-base language model to obtain the data output by the large-base language model, wherein an initial rank value is selected from the multiple variables according to the prior distribution, a corresponding initial low-rank matrix is generated according to the initial rank value and the preset matrix, and the data in the large-base language model is updated according to the initial low-rank matrix. The original weight matrix is used to calculate a first loss value based on the labeled data of the first training sample and the data output by the large-base language model for the first training sample. The large-base language model is then adjusted based on the first loss value to train a fine-tuned model corresponding to the initial rank value. The adjustment of the large-base language model includes keeping the original weight matrix fixed and updating the initial low-rank matrix. The adjustment process of the large-base language model is repeated until all variables are selected and then stopped to obtain the fine-tuned model corresponding to each variable. Based on the performance index and corresponding resource consumption of each fine-tuned model, the expected rank value is determined from the multiple variables.
[0010] In some embodiments of this application, the step of training the large-scale language model based on the second training sample and the target low-rank matrix using the LoRA method to obtain the industrial large-scale language model includes: inputting the second training sample into the large-scale language model to obtain the output data of the large-scale language model, wherein the original weight matrix of the large-scale language model is updated according to the target low-rank matrix, a second loss value is calculated based on the labeled data of the second training sample and the output data of the large-scale language model for the second training sample, and the large-scale language model is adjusted according to the second loss value to train the industrial large-scale language model, wherein the adjustment of the large-scale language model includes keeping the original weight matrix fixed and updating the target low-rank matrix.
[0011] In some embodiments of this application, the method further includes: extracting information from the prompt context or the answer text to obtain a summary of the answer text.
[0012] In some embodiments of this application, the method further includes: extracting key information from the communication content, automatically classifying and / or prioritizing it.
[0013] On the other hand, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the intelligent question-answering method described above.
[0014] On the other hand, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor in an electronic device, implements the intelligent question-answering method. In the intelligent question-answering method provided in this application embodiment, a target question matching the question text is determined through matching, enabling the target question to more accurately reflect the user's true intent. Various types of industrial knowledge in the industrial knowledge base are categorized, so the electronic device can quickly retrieve industrial knowledge matching the question text based on the category of the target question. Since the question text accurately reflects the user's true intent, and the industrial knowledge reflects standard and accurate industrial background knowledge, combining the question text and the retrieved industrial knowledge using prompt word templates ensures that the prompt context not only accurately reflects the user's query needs but also provides rich and accurate domain knowledge for the industrial large language model. Thus, inputting the prompt context into the industrial large language model enhances its semantic analysis, understanding, and reasoning capabilities. Therefore, this application significantly improves question-answering efficiency, reduces labor costs, and enhances the accuracy and practicality of answers, providing strong support for the digital transformation of the industrial sector. Attached Figure Description
[0015] Figure 1 This is a flowchart of an intelligent question-answering method provided in an embodiment of this application.
[0016] Figure 2 This is a flowchart of a method for determining a target problem provided in an embodiment of this application.
[0017] Figure 3 This is a flowchart of a training method for an industrial large language model provided in an embodiment of this application.
[0018] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.
[0020] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.
[0021] This application provides an intelligent question-answering method that can improve question-answering efficiency.
[0022] The intelligent question-answering method provided in this application can be applied to one or more electronic devices, such as servers, computers, mobile phones, laptops, tablets, wearable devices, etc. The server can be a cloud server or a server cluster. This application does not limit the type of electronic device.
[0023] like Figure 1The diagram shown is a flowchart of an intelligent question-answering method provided in an embodiment of this application. Depending on different needs, the order of the steps in this flowchart can be adjusted according to actual requirements, and some steps can be omitted. The intelligent question-answering method is applied to electronic devices, such as… Figure 4 The electronic device 10 shown.
[0024] S11, Obtain the question text input by the user.
[0025] In some embodiments of this application, the electronic device can provide a user interface to facilitate human-computer interaction. The user interface can have various types of controls, such as buttons and text boxes, allowing users to input information. The electronic device can receive the user's input text through the user interface.
[0026] It is understandable that in other implementations, the user's input text can also be obtained via voice.
[0027] S12, Based on the question text, determine the matching target question from the preset question database.
[0028] In some embodiments of this application, the question library includes multiple preset questions, and each preset question corresponds to a vector representation. The preset questions can be questions and data from the industrial field, each preset question has a corresponding category, and the method for generating the vector representation corresponding to each preset question can be found in the description of the question vector generation method below.
[0029] In some embodiments of this application, the electronic device determines a matching target question from a preset question library based on question text by: vectorizing the question text to obtain a question vector, and determining the target question from the multiple preset questions based on the matching of the question vector with multiple vector representations.
[0030] The electronic device can invoke an embedding model to vectorize the question text and obtain a question vector. The embedding model can be Word2Vec, BGE, or BERT, among others.
[0031] Considering that question text may contain semantic ambiguity and non-standard professional terminology, this embodiment converts the question text into question vectors to facilitate the extraction of semantic features. By performing vector matching on the question vectors, target questions that match the question text are determined, enabling the target questions to more accurately reflect the user's true intent.
[0032] In other embodiments, the question text can be segmented using a word segmentation algorithm before being vectorized. For example, word segmentation algorithms include, but are not limited to, rule-based segmentation, statistical segmentation, and deep learning-based segmentation.
[0033] For example, an electronic device can use an approximate nearest neighbor search algorithm to match a problem vector with multiple vector representations, thereby determining a target problem from a plurality of preset problems. This application does not limit the type of approximate nearest neighbor search algorithm. For example, an approximate nearest neighbor search algorithm could be an inverted file algorithm based on flat quantization (IVF-Flat), etc.
[0034] In some embodiments of this application, an electronic device can retrieve relevant information from an external knowledge base based on a target question using Retrieval-Augmented Generation (RAG) technology and generate accurate answer text. This method retrieves relevant information and data from a knowledge base in the relevant domain before answering the question, and then uses a Large Language Model (LLM) based on this information to generate more accurate and relevant answer text. The method for generating accurate answer text using RAG technology will be described in detail below.
[0035] S13, based on the category of the target question, retrieve industrial knowledge that matches the question text from the preset industrial knowledge base.
[0036] In some embodiments of this application, the industrial knowledge base can be a comprehensive database in the industrial field, which can include multi-category and multi-level industrial knowledge, covering multiple sub-fields of the industrial field. For example, the industrial knowledge base can include knowledge resources such as production data, equipment information, technical documents, industry specifications and standards, and historical industrial cases. Among them, production data can include production process flow and production parameters, equipment information can include operation logs, fault diagnosis records and solutions, equipment maintenance methods, historical operating status and performance evaluation, etc., and technical documents can include various technical details, innovative methods and practical experience.
[0037] In some embodiments of this application, the electronic device can accept industrial knowledge input by the user through a user interface, perform data cleaning on the industrial knowledge, classify the cleaned industrial knowledge, and construct an industrial knowledge base based on each type of industrial knowledge and its corresponding category. The data cleaning of industrial knowledge includes, but is not limited to: deleting abnormal data, standardizing data formats, and handling missing values.
[0038] In some embodiments, electronic devices can directly categorize industrial knowledge according to the type of industrial knowledge, thereby enabling the construction of an industrial knowledge base.
[0039] In other embodiments, the electronic device may use an embedding model to vectorize each type of industrial knowledge, so that each type of industrial knowledge can be stored in an industrial knowledge base in the form of vectors.
[0040] In some embodiments of this application, the electronic device can use the industrial knowledge corresponding to the category of the target problem in an industrial knowledge base as the industrial knowledge to match the question text. For example, if the category of the target problem is "production safety", the electronic device can use the regulations and other knowledge corresponding to the category "production safety" in the industrial knowledge base as the industrial knowledge to match the question text.
[0041] In this embodiment, since various types of industrial knowledge in the industrial knowledge base are categorized according to categories, the electronic device can quickly retrieve industrial knowledge that matches the question text based on the category of the target question.
[0042] S14, Based on industry knowledge and the problem text, generate a prompt context.
[0043] In some embodiments of this application, electronic devices can combine industry knowledge with question text using prompt word templates to obtain prompt context.
[0044] The prompt template can include the question text and the corresponding information bar for industrial knowledge. Electronic devices can obtain the prompt context by filling the corresponding information bar with the retrieved industrial knowledge and question text.
[0045] In addition to the question text and the information bar corresponding to industry knowledge, the prompt word template can also include the output format of data extraction tasks, examples, and answer text. The data extraction task can be used to prompt the large language model to extract data, and the examples can be used to provide reference for the large language model. The prompt context can be used as prompt words to prompt the large language model for reference examples, and the corresponding format of answer text is output according to the data extraction task. The output format of data extraction tasks, examples, and answer text in the prompt word template can be customized, and this application does not impose any restrictions on this.
[0046] In this embodiment, since the question text can truly reflect the user's real intent and the industrial knowledge can reflect standard and accurate industrial background knowledge, by combining the question text and the retrieved industrial knowledge with the prompt word template, the prompt context can not only truly and accurately reflect the user's query needs, but also provide rich and accurate industrial domain knowledge for the large language model.
[0047] In other embodiments of this application, the electronic device may also utilize a combination of industry knowledge and target questions as prompt context using prompt word templates.
[0048] In this embodiment, since the target question is standardized, combining the target question with the retrieved industry knowledge through prompt word templates can improve the professionalism, accuracy, and richness of the prompt context.
[0049] S15 uses an industrial large language model to reason about the context of the prompt and generates the answer text corresponding to the question text.
[0050] In some embodiments of this application, the electronic device can acquire industrial corpora as training samples. Based on the training samples, a large-scale industrial language model is trained using a Bayesian algorithm and the low-rank adaptive fine-tuning LoRA method. The industrial corpora include, but are not limited to, equipment identifiers, technical terms, and exception codes. The large-scale industrial language model can be a Transformer model, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer) model, or a Qwen model, etc.
[0051] In some embodiments of this application, the answer text may include operational procedures, decision suggestions, early warning information, and data analysis results. For example, following the above embodiments, if the retrieved industrial knowledge is a regulation or rule of production safety, the answer text output by the industrial big data language model may include key information from the regulation or rule of production safety. In this embodiment, inputting the contextual prompts into the industrial big language model can enhance the semantic analysis, understanding, and reasoning capabilities of the industrial big language model, thereby ensuring the accuracy of the answer text and improving question-and-answer efficiency.
[0052] In other embodiments, the electronic device may utilize an industrial large language model to extract information from the prompt context or answer text to obtain a summary of the answer text.
[0053] In other embodiments, the electronic device may display the answer text and corresponding summary on the user interface for the user to view.
[0054] In other embodiments, natural language generation technology and key information extraction algorithms can be used to automatically classify and prioritize key information based on the importance and urgency of the communication content, providing decision-makers with timely and targeted decision support suggestions. For example, automatic classification and / or prioritization can be performed based on the type of event, the object, or the time, and key information can be prioritized to provide decision-makers with timely and targeted decision support suggestions.
[0055] In other embodiments, the electronic device can receive user-input instructions and feedback information through a user interface, thereby enabling it to intervene in and adjust the decisions of the industrial large language model.
[0056] In other embodiments, the electronic device can verify the accuracy of the answer text output by the industrial large language model. When it is determined that the answer text output by the industrial large language model is incorrect, the electronic device can determine that the prompt context and the industrial knowledge retrieved from the industrial knowledge base are inaccurate, and can obtain industrial knowledge through technologies such as web crawling. The obtained industrial knowledge, or the obtained industrial knowledge after vectorization, can be stored in the industrial knowledge base to achieve updates and optimization of the industrial knowledge base. The electronic device can verify the accuracy of the answer text using various methods, and this application does not limit the verification method. For example, it can determine the accuracy by developers periodically checking and receiving user feedback, or by comparing the similarity between the question text and the answer text. In some embodiments of this application, the electronic device can utilize Docker container technology to encapsulate and deploy the embedded model and the industrial large language model, and utilize Docker Swarm technology for cluster management. This not only allows the embedded model and the industrial large language model to run as independent services but also improves resource utilization. When implementing cluster management using Docker Swarm technology, image signing and container isolation measures can be set to ensure the operational security of the embedded model and the industrial large language model. During the operation of the industrial large language model, the electronic device can utilize a Graphics Processing Unit (GPU) to accelerate the inference speed of the industrial large language model.
[0057] In the intelligent question-answering method provided in this application embodiment, a target question matching the question text is determined through matching, enabling the target question to more accurately reflect the user's true intent. Various types of industrial knowledge in the industrial knowledge base are categorized, so the electronic device can quickly retrieve industrial knowledge matching the question text based on the category of the target question. Since the question text accurately reflects the user's true intent, and the industrial knowledge reflects standard and accurate industrial background knowledge, combining the question text with the retrieved industrial knowledge using prompt word templates ensures that the prompt context not only accurately reflects the user's query needs but also provides rich and accurate domain knowledge for the industrial big data language model. Thus, inputting the prompt context into the industrial big data language model enhances its semantic analysis, understanding, and reasoning capabilities, significantly improving question-answering efficiency, reducing labor costs, and increasing the accuracy and practicality of the answers, providing strong support for the digital transformation of the industrial sector.
[0058] For example, such as Figure 2 The diagram shown is a flowchart of a method for determining a target problem according to an embodiment of this application, including the following steps: S121, Based on the clustering of the multiple vector representations, determine multiple clusters.
[0059] In some embodiments of this application, the electronic device can identify multiple clusters by exemplifying the multiple vector representations using various methods. This application does not limit the clustering methods. Examples include K-means clustering and density-based spatial clustering of applications with noise (DBSCAN). Each cluster includes at least one vector representation.
[0060] In some embodiments of this application, the electronic device can generate a corresponding index for each cluster and all vector representations within the cluster, and construct an inverted index based on the index of each cluster and the index of all vector representations within the cluster, so as to map each vector representation to its respective cluster.
[0061] S122, calculate the distance between the problem vector and the centroid of each cluster, and select candidate clusters from the plurality of clusters based on the distance.
[0062] In some embodiments of this application, the centroid of each cluster can be the center of the cluster or a representative vector generated from the vector representations within that cluster. For example, the electronic device can perform a weighted average operation on all vector representations within each cluster based on the weights corresponding to each vector representation in that cluster to obtain a representative vector representing the centroid. The weight corresponding to each vector representation is determined by the distance between that vector representation and the center of the cluster in space. Alternatively, the electronic device can calculate the median of all vector representations within each cluster in each feature dimension, and then combine all the calculated medians into a single vector as the representative vector of the centroid of that cluster.
[0063] In some embodiments of this application, the electronic device can select the cluster corresponding to the smallest distance as a candidate cluster. The distance between the problem vector and the centroid of each cluster can be Euclidean distance, Manhattan distance, etc.
[0064] S123, Based on the similarity between the problem vector and the vector representations in the candidate cluster, determine the target problem from the preset problems corresponding to the vector representations in the candidate cluster.
[0065] In some embodiments of this application, the electronic device can calculate the similarity between the question vector and each vector representation in the candidate cluster using various methods. For example, the electronic device can use the Euclidean distance, Manhattan distance, or other distances between the question vector and each vector representation in the candidate cluster as the corresponding similarity. The electronic device can select the preset question to which the vector representation with the highest similarity in the candidate cluster belongs as the target question.
[0066] In this embodiment, since each vector representation is mapped to its respective cluster, the electronic device can quickly retrieve the cluster most similar to the problem vector as a candidate cluster based on the candidate cluster and the index of each vector representation in the candidate cluster.
[0067] For example, following the above embodiments, the training samples from the industrial corpus may include a first training sample and a second training sample, wherein the number of samples in the first training sample is less than the number of samples in the second training sample. Electronic devices can train a large-scale language model based on the first and second training samples to obtain an industrial large-scale language model. Figure 3 The diagram shown is a flowchart of a training method for an industrial large-scale language model provided in an embodiment of this application, including the following steps: S151, based on the large-base language model and the first training sample, uses the Bayesian algorithm to determine the expected rank value.
[0068] In some embodiments of this application, the electronic device can utilize the Bayesian algorithm to construct a prior distribution of multiple variables, adjust the base large language model based on the first training sample and the prior distribution to obtain a fine-tuned model corresponding to each variable, and determine the expected rank value from the multiple variables according to the performance index and corresponding resource consumption of each fine-tuned model.
[0069] The expected rank is also known as the average rank, and the multiple variables can be random variables. Performance metrics for fine-tuning the model can include precision, recall, the harmonic mean of precision and recall, and the AUC score. The harmonic mean of precision and recall is also known as the F1 score. Resources can include time resources and the computing power of the electronic device's processor. For example, resource consumption can include model training time and the computing power consumed by the electronic device.
[0070] Electronic devices can use the variables corresponding to the performance indicators and corresponding resource consumption of a fine-tuned model that meet a preset optimization objective as the expected rank value. The preset optimization objective can be customized. For example, the preset optimization objective could be to achieve optimal performance of the fine-tuned model within a preset computing power budget, or to minimize the computing power consumption of the electronic device's processor while meeting performance requirements.
[0071] In this embodiment, the performance of the corresponding fine-tuning model can be tested based on the performance indicators and the corresponding resource consumption. Based on the experimental verification, the desired rank value can be selected from the multiple variables, which can maintain high model performance while reducing computing power consumption.
[0072] For example, an electronic device adjusts a large-base language model based on a first training sample and a prior distribution to obtain a fine-tuned model corresponding to each variable. This includes: inputting the first training sample into the large-base language model to obtain the data output by the large-base language model; selecting an initial rank value from the plurality of variables according to the prior distribution; generating a corresponding initial low-rank matrix based on the initial rank value and a preset matrix; updating the original weight matrix based on the initial low-rank matrix; calculating a first loss value based on the labeled data of the first training sample and the data output by the large-base language model for the first training sample; adjusting the large-base language model based on the first loss value; and training to obtain a fine-tuned model corresponding to the initial rank value. The adjustment of the large-base language model includes keeping the original weight matrix fixed and updating the initial low-rank matrix. The adjustment process for the large-base language model is repeated until all the plurality of variables have been selected, and then the fine-tuned model corresponding to each variable is obtained.
[0073] The prior distribution describes the probability of an event occurring under different values of k, where the event can be the act of choosing k as the expected rank. Taking the Poisson distribution as an example, the prior distribution can be referred to by the following formula (1): (1) in, Represents a random variable that is greater than or equal to zero. This represents the probability of choosing the value k as the expected rank. This represents the expected rank value.
[0074] An initial rank value can be selected from the plurality of variables based on the probability corresponding to each variable in the prior distribution. For example, the variables can be selected as initial rank values according to their probability order from high to low. The preset matrix can be custom-defined, or it can be the original weight matrix of a large-base language model. The electronic device can perform low-rank decomposition on the preset matrix based on each selected initial rank value to obtain an initial low-rank matrix. The initial low-rank matrix includes a first initial low-rank matrix and a second initial low-rank matrix.
[0075] The electronic device can determine the updated weight matrix based on the initial low-rank matrix and the original weight matrix, and replace the original weight matrix in the large-base language model with the updated weight matrix. For example, the calculation method of the updated weight matrix can refer to the following formulas (2)~(3): (2) (3) in, This represents the updated weight matrix. Represents the original weight matrix. Denotes the first initial low-rank matrix. Let represent the second initial low-rank matrix.
[0076] For example, if the performance metric is accuracy, the electronic device can use the variable corresponding to the highest accuracy as the expected rank. Alternatively, if the performance metric is F1 score, the electronic device can use the variable corresponding to the highest F1 score as the expected rank.
[0077] The labeled data of the first training sample can be category labels, etc. For example, the first loss value can be cross-entropy loss, etc., and this application does not limit the type of the first loss value. The electronic device can adjust the large-base language model until the first loss value meets a preset condition, thus training a fine-tuned model. The preset condition can be customized, and this application does not limit it. For example, the preset condition can be that the first loss value is within a preset numerical range or that the first loss value is less than a preset threshold.
[0078] The selection of the aforementioned variables is merely an example of stopping the adjustment of the base-large language model; in practice, it is not limited to this.
[0079] In other embodiments of this application, the probability of each variable being selected as the expected rank value is related to its corresponding probability in the prior distribution. For example, the higher the probability of any variable in the prior distribution, the greater the probability that the variable will be selected as the expected rank value; conversely, the lower the probability of any variable in the prior distribution, the less likely the variable will be selected as the expected rank value.
[0080] In this embodiment, initial rank values are selected from the multiple variables based on prior distributions. This simulates the scenario of selecting different variables as desired rank values. The original weight matrix of the large-base language model is updated using the initial low-rank matrix of each selected initial rank value. This allows each variable to correspond to a fine-tuning model, facilitating the subsequent selection of desired rank values from the multiple variables based on the performance metrics and corresponding resource consumption of the fine-tuning model. By freezing the original weight matrix of the large-base language model and updating only the initial low-rank matrix, the number of trainable parameters and the overhead of gradient calculation are significantly reduced. This achieves efficient fine-tuning of the large-base language model while reducing computational power consumption.
[0081] S152, determine the target low-rank matrix based on the preset matrix and the expected rank value.
[0082] In some embodiments of this application, the electronic device can perform low-rank decomposition on a preset matrix based on a desired rank value to obtain a target low-rank matrix. The target low-rank matrix may include a first target low-rank matrix and a second target low-rank matrix.
[0083] S153, based on the second training samples and the target low-rank matrix, uses the LoRA method to train the base large language model to obtain the industrial large language model.
[0084] In some embodiments of this application, the electronic device trains a large-scale language model based on a second training sample and a target low-rank matrix using the LoRA method to obtain an industrial large-scale language model, including: inputting the second training sample into the large-scale language model to obtain the output data of the large-scale language model, wherein the original weight matrix of the large-scale language model is updated according to the target low-rank matrix, a second loss value is calculated based on the labeled data of the second training sample and the output data of the large-scale language model for the second training sample, and the large-scale language model is adjusted according to the second loss value to train the industrial large-scale language model, wherein the adjustment of the large-scale language model includes keeping the original weight matrix fixed and updating the target low-rank matrix.
[0085] The target low-rank matrix includes a first target low-rank matrix and a second target low-rank matrix. The method for updating the original weight matrix based on the first target low-rank matrix and the second target low-rank matrix can be found in the above description of updating the original weight matrix based on the first initial low-rank matrix and the second initial low-rank matrix.
[0086] The labeled data of the second training sample can be category labels, etc. This application does not restrict the type of the second loss value; for example, the second loss value can be the cross-entropy loss value. The process of adjusting the large-base language model based on the second loss value can be found in step S151, which describes adjusting the large-base language model using the first loss value.
[0087] Given the enormous number of parameters in large-scale language models—for example, the GPT series models can have tens of billions of parameters—directly fine-tuning all parameters of a large-scale language model would not only require storing and updating massive gradients, leading to low training efficiency, but also consume a large amount of computing resources due to the huge computational and storage overhead. Therefore, this embodiment freezes the original weight matrix of the large-scale language model and updates only the target low-rank matrix, significantly reducing the number of trainable parameters and the overhead of gradient calculation. This achieves efficient fine-tuning of the large-scale language model while reducing computational consumption. Furthermore, updating the original weight matrix using the target low-rank matrix corresponding to the expected rank value not only significantly reduces the computational consumption during the operation of the large-scale language model (e.g., reducing the computational cost of matrix multiplication), but also ensures that the performance of the large language model is not significantly affected.
[0088] like Figure 4 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. The electronic device 10 can be a computer, mobile phone, tablet computer, laptop computer, server, or other similar device. This application embodiment does not impose any restrictions on the specific type of the electronic device 10.
[0089] exist Figure 4 The electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the input / output interface 104 via the bus 105.
[0090] Communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, frequency modulation (FM), near field communication (NFC), and infrared (IR).
[0091] Memory 102 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 103, and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.
[0092] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 103. Non-volatile memory can include disk storage devices and flash memory.
[0093] Memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 103. The one or more computer programs include multiple instructions that, when executed by processor 103, enable an intelligent question-answering method to be executed on electronic device 10.
[0094] In other embodiments, such as Figure 4 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.
[0095] Processor 103 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.
[0096] The processor 103 provides computing and control capabilities; for example, the processor 103 is used to execute computer programs stored in the memory 102 to implement the intelligent question-answering method described above.
[0097] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input / output devices, such as a mouse, keyboard, touch device, display screen, etc., so that users can enter information or visualize information.
[0098] Bus 105 is used at least to provide a channel for communication between communication modules 101, memory 102, processor 103, and input / output interface 104 in electronic device 10.
[0099] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0100] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can refer to the methods in the above embodiments of this application.
[0101] The computer-readable storage medium can be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. Alternatively, the computer-readable storage medium can be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., installed on the electronic device.
[0102] In some embodiments, a computer-readable storage medium may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the electronic device, etc.
[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0105] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0106] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0107] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. An intelligent question-answering method, characterized in that, The method includes: Get the question text entered by the user; Based on the question text, a matching target question is determined from a preset question database; Based on the category of the target question, retrieve industrial knowledge that matches the question text from a preset industrial knowledge base; Based on the aforementioned industry knowledge and the question text, a prompt context is generated; The industrial large language model is used to reason about the context of the prompt to generate the answer text corresponding to the question text.
2. The intelligent question-answering method as described in claim 1, characterized in that, The question database includes multiple preset questions, and each preset question corresponds to a vector representation. The step of determining the matching target question from the preset question database based on the question text includes: The question text is vectorized to obtain the question vector; The target problem is determined from the plurality of preset problems by matching the problem vector with multiple vector representations.
3. The intelligent question-answering method as described in claim 2, characterized in that, The step of determining the target question from the plurality of preset questions based on matching the question vector with the vector representation includes: Multiple clusters are determined based on the clustering of the multiple vector representations; Calculate the distance between the problem vector and the centroid of each cluster, and select candidate clusters from the plurality of clusters based on the distance; Based on the similarity between the question vector and the vector representations in the candidate cluster, the target question is determined from the preset questions corresponding to the vector representations in the candidate cluster.
4. The intelligent question-answering method as described in claim 1, characterized in that, The training method for the industrial large language model includes: The industrial corpus is used as training samples, and the industrial corpus includes one or more of equipment identifiers, technical terms, and exception codes. Based on the training samples, the industrial large language model is obtained by training the base large language model using the Bayesian algorithm and the low-rank adaptive fine-tuning LoRA method.
5. The intelligent question-answering method as described in claim 4, characterized in that, The training samples include a first training sample and a second training sample, wherein the number of samples in the first training sample is less than the number of samples in the second training sample. The process of training the large-scale language model using the Bayesian algorithm and the low-rank adaptive fine-tuning LoRA method to obtain the industrial large-scale language model includes: Based on the aforementioned large-base language model and the first training sample, the expected rank value is determined using the Bayesian algorithm. Based on the preset matrix and the desired rank value, determine the target low-rank matrix; Based on the second training samples and the target low-rank matrix, the industrial large language model is obtained by training the basis large language model using the LoRA method.
6. The intelligent question-answering method as described in claim 5, characterized in that, The determination of the expected rank value based on the large-base language model and the first training sample, using the Bayesian algorithm, includes: Using the Bayesian algorithm, a prior distribution for multiple variables is constructed; Based on the first training samples and the prior distribution, the base-large language model is adjusted, including: The first training sample is input into the large-base language model to obtain the data output by the large-base language model. An initial rank value is selected from the multiple variables according to the prior distribution. A corresponding initial low-rank matrix is generated according to the initial rank value and the preset matrix. The original weight matrix in the large-base language model is updated according to the initial low-rank matrix. The first loss value is calculated based on the labeled data of the first training sample and the data output by the large-base language model for the first training sample; The large-base language model is adjusted according to the first loss value to train a fine-tuned model corresponding to the initial rank value. The adjustment of the large-base language model includes keeping the original weight matrix unchanged and updating the initial low-rank matrix. Repeat the adjustment process for the base large language model until all the variables have been selected, and then stop to obtain the fine-tuned model corresponding to each variable; The desired rank value is determined from the multiple variables based on the performance metrics and corresponding resource consumption of each fine-tuning model.
7. The intelligent question-answering method as described in claim 5, characterized in that, The step of training the large-scale industrial language model using the LoRA method based on the second training samples and the target low-rank matrix includes: The second training sample is input into the large-base language model to obtain the output data of the large-base language model, wherein the original weight matrix of the large-base language model is updated according to the target low-rank matrix; The second loss value is calculated based on the labeled data of the second training sample and the output data of the large-base language model for the second training sample; The large-base language model is adjusted according to the second loss value to train the industrial large-base language model, wherein the adjustment of the large-base language model includes keeping the original weight matrix unchanged and updating the target low-rank matrix.
8. The intelligent question-answering method as described in claim 1, characterized in that, The method further includes: Information is extracted from the context of the prompt or the answer text to obtain a summary of the answer text.
9. The intelligent question-answering method as described in claim 1, characterized in that, The method further includes: Extract key information from communication content and automatically classify and / or prioritize it.
10. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the intelligent question-answering method as described in any one of claims 1 to 9.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which, when executed by a processor in an electronic device, implements the intelligent question-answering method as described in any one of claims 1 to 9.