Vertical domain information processing method and device based on large language model

Abstract

The invention discloses a vertical domain information processing method and device based on a large language model, and the method comprises the steps: receiving a user request, and determining knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm; determining a cue word template according to a user request and / or knowledge information, filling the cue word template with the knowledge information, and encoding to obtain a model input matrix; inputting the model input matrix into the large language model to obtain a middle matrix and an output matrix, and decoding the output matrix to obtain an output answer; determining an information sub-matrix corresponding to the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix; and according to the information sub-matrix, judging whether the first type of knowledge information in the vertical domain knowledge base is utilized in the output answer. According to the technical scheme, the problem that a general large language model is poor in processing effect due to the fact that conventional training data does not relate to vertical field information can be solved.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent electric vehicles and natural language processing, and in particular to a vertical field information processing method and device based on a large language model. Background Art

[0002] At present, as the functions of smart electric vehicles become increasingly powerful, the complexity of automobile products is also increasing. Since a large number of new technologies, new processes, and new components are used in the process of automobile R&D and production, there is also a large amount of data and information in the technical service field related to these technologies that needs to be processed to meet the needs of product development, quality control, and user services.

[0003] Given the professional nature of the above data and information, their processing is mainly completed by professional engineers and customer service personnel. By summarizing, screening, and retelling the original technical data, the technical information is converted into text or other data forms required for the target scenario, and then delivered to user services or technical personnel in the target department for use. The entire process consumes huge communication and processing costs, and the accuracy of information processing is often insufficient due to low communication efficiency.

[0004] With the rise of large language models based on transformer network architecture, a general artificial intelligence model for text processing with high flexibility, understanding ability and scenario generalization ability has come into people's eyes. Different from general scenarios, due to information confidentiality and data timeliness, the training corpus of large language models generally does not include text and data in the field of electric vehicle R&D technology, let alone enterprise-level confidential document information, so the information processing performance in vertical field scenarios such as smart electric vehicles is not good. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a method and device for vertical domain information processing based on a large language model that overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a method for processing vertical domain information based on a large language model is provided, the method comprising:

[0007] Receive a user request, and determine the knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm;

[0008] Determine a prompt word template according to the user request and / or the knowledge information, fill the knowledge information into the prompt word template according to a preset format, and encode the filled prompt word template to obtain a model input matrix;

[0009] Inputting the model input matrix into a large language model to obtain an intermediate matrix and an output matrix, and decoding the output matrix to obtain an output answer;

[0010] Determining, according to the position of the knowledge information in the model input matrix, an information submatrix corresponding to the knowledge information in the intermediate matrix;

[0011] It is determined whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

[0012] In some implementations, assuming that the model input vector matrix is ​​an N*M matrix, and the output matrix is ​​a K*M matrix, where N is the number of word units in the prompt word template, K is the number of word units in the model output answer, and M is the expression length of each word unit, the model input matrix is ​​input into the large language model, and the intermediate matrix obtained includes:

[0013] Obtain K N*N weight matrices obtained by K operations, calculate the arithmetic mean of the K N*N weight matrices, obtain an arithmetic mean N*N matrix, and use the arithmetic mean N*N matrix as an intermediate matrix;

[0014] Determining the information submatrix of the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix includes:

[0015] Determine the vector position of the knowledge information in the N*M matrix according to the position of the prompt word template filled with the knowledge information;

[0016] An information sub-matrix is ​​determined from the arithmetic mean N*N matrix according to the vector position.

[0017] In some implementations, judging whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix includes:

[0018] Calculating the arithmetic mean of each value in the information submatrix, and determining whether the arithmetic mean is greater than a preset empirical threshold, and if it is not less than the empirical threshold, determining that the output result utilizes the first type of knowledge information; or,

[0019] Encoding the knowledge information to obtain a knowledge information matrix, and calculating a similarity value between the information sub-matrix and the knowledge information matrix;

[0020] The similarity value is compared with a preset similarity threshold, and if the similarity threshold is greater than or equal to the similarity threshold, it is determined that the output result uses the first type of knowledge information.

[0021] In some implementations, inputting the model input matrix into a large language model to obtain an intermediate matrix and an output matrix, and encoding the output matrix to obtain an output answer comprises:

[0022] Convert each word element vector in the output matrix into a unit vector whose sum is 1;

[0023] Calculate the distribution entropy value of each unit vector;

[0024] Calculate the arithmetic mean of the distribution entropy values, and use the arithmetic mean as the answer confidence;

[0025] The answer confidence value is compared with a preset confidence threshold, and the answer is output if the confidence value is less than or equal to the confidence threshold.

[0026] In some embodiments, the method further comprises:

[0027] If it is determined that the output answer does not utilize the first type of knowledge information or the confidence value of the answer is greater than a confidence threshold, the output result is output to a human expert so that the human expert can provide modification suggestions.

[0028] In some embodiments, the method further comprises:

[0029] The large language model is trained using knowledge information in a vertical domain knowledge base.

[0030] In some embodiments, the vertical field is the field of smart electric vehicles, and the first category of knowledge information is smart electric vehicle knowledge, including at least one of the following: new product development information, user vehicle service information, production line upgrade and transformation information, vehicle failure causes and solutions information, or user service standard operating procedures.

[0031] According to another aspect of the present invention, a vertical domain information processing device based on a large language model is provided, the device comprising:

[0032] A similarity calculation module, adapted to receive a user request and determine the knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm;

[0033] A filling encoding module, adapted to determine a prompt word template according to the user request and / or the knowledge information, fill the knowledge information into the prompt word template according to a preset format, and encode the filled prompt word template to obtain a model input matrix;

[0034] A model operation module, adapted to input the model input matrix into a large language model, obtain an intermediate matrix and an output matrix, and decode the output matrix to obtain an output answer;

[0035] a position determination module, adapted to determine an information submatrix corresponding to the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix;

[0036] The effect evaluation module is adapted to determine whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

[0037] According to another aspect of the present invention, a computing device is provided, comprising: a processor and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the vertical domain information processing method based on a large language model according to any one of the above-mentioned embodiments.

[0038] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the vertical field information processing method based on a large language model according to any of the above items is implemented.

[0039] In summary, according to the vertical field information processing method based on the large language model disclosed in the present invention, on the one hand, the large language model is used to liberate the energy of data engineers and user service personnel and reduce the cost of information processing; on the other hand, by introducing the effect evaluation step, the ability of the large language model to accurately use the vertical field knowledge base is improved, thereby solving the problem of inaccurate information processing results caused by the large language model not including vertical field knowledge information in conventional training data, so that the general large language model can still achieve a high information processing accuracy rate when trained in the absence of vertical field knowledge.

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0042] Figure 1 A schematic diagram of a process of a vertical domain information processing method based on a large language model according to some embodiments of the present invention is shown;

[0043] Figure 2 A schematic diagram of a large language model operation process according to some embodiments of the present invention is shown;

[0044] Figure 3 A schematic diagram of a process flow of effect evaluation according to some embodiments of the present invention is shown;

[0045] Figure 4 A schematic diagram showing the overall process of vertical field information processing according to some embodiments of the present invention is shown;

[0046] Figure 5 A schematic diagram of the structure of a vertical field information processing device based on a large language model according to some embodiments of the present invention is shown;

[0047] Figure 6 A schematic diagram of the structure of a computing device according to some embodiments of the present invention is shown;

[0048] Figure 7 A structural schematic diagram showing the connection relationship between various devices according to some embodiments of the present invention. DETAILED DESCRIPTION

[0049] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0050] Figure 1 A flowchart of a vertical domain information processing method based on a large language model according to an embodiment of the present invention is shown, and the method includes the following steps:

[0051] Step S110, receiving a user request, and determining knowledge information closest to the user request from a vertical domain knowledge base according to the user request by using a semantic similarity algorithm;

[0052] Among them, the semantic similarity algorithm includes but is not limited to the cosine similarity algorithm or the word2vec algorithm, etc. The vertical field knowledge base is a specific knowledge base constructed according to the characteristics of a certain field, and there are significant differences from the corpus used for large language model training. The number of knowledge information can be one or more.

[0053] Step S120, determining a prompt word template according to the user request and / or the knowledge information, filling the knowledge information into the prompt word template according to a preset format, and encoding the filled prompt word template to obtain a model input matrix;

[0054] The prompt word template may include user requests, system prompt words and knowledge information in a vertical domain knowledge base, and the system prompt words and knowledge information may be retrieved and determined according to the user request. Figure 3 A complete prompt word template is shown, the prompt word part: You are an automobile after-sales service staff, please answer the user's question based on the user's question and relevant knowledge; user request: How long does it take to maintain my vehicle; knowledge information of the vertical domain knowledge base: general maintenance takes about four hours.

[0055] It should be pointed out that the general training result output of the large language model may be: The general maintenance takes about three hours, which is different from the knowledge information in the vertical domain knowledge base, and the embodiment of the present invention is to guide or evaluate the output results to effectively utilize the knowledge information in the vertical domain knowledge base;

[0056] Step S130, inputting the model input matrix into the large language model to obtain an intermediate matrix and an output matrix, and decoding the output matrix to obtain an output answer;

[0057] The intermediate matrix may be a weight matrix output by a certain layer in a large language model. The technical concept of the embodiment of the present invention is to use the characteristics or properties of the intermediate matrix output by the large language model to analyze whether the final output result includes or uses the information of the vertical domain knowledge base.

[0058] Step S140, determining an information sub-matrix corresponding to the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix;

[0059] In order to evaluate whether the intermediate matrix includes knowledge information, the information submatrix corresponding to the knowledge information in the intermediate matrix is ​​first obtained;

[0060] Step S150, judging whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

[0061] When using the information sub-matrix for judgment, the characteristics of the information sub-matrix itself can be considered, and the correlation between the information sub-matrix and other information vectors or matrices can also be considered.

[0062] In summary, according to the above embodiments of the present invention, on the one hand, the large language model is used to free up the energy of data engineers and user service personnel and reduce information processing costs; on the other hand, by introducing steps such as effect evaluation, the ability of the large language model to accurately use the vertical domain knowledge base is improved, thereby solving the problem of inaccurate information processing results of the large language model due to the fact that the conventional training data does not include vertical domain knowledge information, so that the general large language model can still achieve a high information processing accuracy rate when trained in the absence of vertical domain knowledge.

[0063] It should be noted that the large language models (LLMs) in the above embodiments are a type of deep learning model with a large number of parameters. In the field of natural language processing, they learn language patterns, grammar and semantics by processing a large amount of text data, so as to understand and generate human language. Large language models include but are not limited to neural network models based on the Transformer architecture.

[0064] Combination Figure 2 , Figure 3 ,and Figure 4 As shown, the embodiment of the present invention adopts the Tokenizer encoder and decoder in model training and model application. Token refers to the basic unit of text, that is, word unit. In natural language processing, Token is the smallest semantic unit that uses numbers to represent word units in language models. It is the process of breaking down text into the smallest units called Tokenization. Token can be a character, word, phrase or sentence, etc., depending on the type of language and the processing method. Tokenizer is a tool or algorithm that divides text into multiple Tokens. It is responsible for dividing the original text into a sequence of Tokens for subsequent processing and analysis. There are many types of Tokenizers, including character-based Tokenizers and word-based Tokenizers, which are suitable for different languages ​​and processing requirements. In order to realize large language model training and application, the segmented word units are further encoded to obtain vectors or matrices.

[0065] In some embodiments, in combination Figure 2 and Figure 3 As shown, assuming that the model input vector matrix is ​​an N*M matrix, and the output matrix is ​​a K*M matrix, where N is the number of word units in the prompt word template, K is the number of word units in the model output answer, M is the expression length of each word unit, and K, M, and N are positive integers greater than 1; then in step S130, the model input matrix is ​​input into the large language model, and the intermediate matrix obtained includes:

[0066] Obtain K N*N weight matrices obtained by K operations, calculate the arithmetic mean of the K N*N weight matrices, obtain an arithmetic mean N*N matrix, and use the arithmetic mean N*N matrix as an intermediate matrix.

[0067] It should be pointed out that in order to obtain the output of K words, the large language model needs to perform K operations, and obtain the weight matrices of N*N, (N+1)*(N+1), (N+2)*(N+2)…(N+K-1)*(N+K-1) in sequence. Starting from the second matrix, take the N*N submatrix in the upper left corner to obtain K N*N weight matrices. Then, calculate their arithmetic average matrix N*N and use the arithmetic average matrix N*N as the intermediate matrix.

[0068] Furthermore, step S140 determines the information submatrix of the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix, including:

[0069] Determine the vector position of the knowledge information in the N*M matrix according to the position of the prompt word template filled with the knowledge information;

[0070] An information sub-matrix is ​​determined from the arithmetic mean N*N matrix according to the vector position.

[0071] Combination Figure 2 As shown, in this embodiment, the model intermediate value 1 can be the position information of the knowledge information in the N*M matrix, which can be determined according to the position of the knowledge information in the prompt word template. For example, in the above template example, the general maintenance requires four hours at the end of the prompt word template, the position is N_1~N, N_1 is a value less than N, and the difference between N and N_1 is the word length of the knowledge information. The model intermediate value 2 is the intermediate matrix, and the (N_1~N)*N of the intermediate matrix is ​​the information sub-matrix. In addition, Figure 2-Figure 4 In the process, the operation of the large language model is regarded as the service agent, and the judgment of whether to use the knowledge information is regarded as the effect evaluation agent.

[0072] It should be pointed out that the above-mentioned knowledge information is placed at the end of the prompt word template for example only, and it can also be placed in the middle or the beginning of the prompt word template.

[0073] Therefore, when the intermediate matrix is ​​determined to be an arithmetic mean N*N matrix, the (N_1~N)*N matrix of the arithmetic mean matrix N*N is the information sub-matrix.

[0074] In some embodiments, for step S150, how to determine whether the output answer uses the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix, two methods for determining the first type of knowledge information are provided. Specifically, the first method includes:

[0075] Combination Figure 3 As shown, the arithmetic mean of each value in the information submatrix is ​​obtained, and it is determined whether the arithmetic mean is greater than a preset empirical threshold. If it is not less than the empirical threshold, it is determined that the output result uses the first type of knowledge information.

[0076] In this method, whether knowledge information is adopted is judged according to the size of the arithmetic mean of the information sub-matrix. The empirical threshold can be determined according to experiments or experience. Moreover, through experimental verification, when the arithmetic mean is greater than or equal to the empirical threshold, it indicates that the output result has a high probability of utilizing the first type of knowledge information.

[0077] The second method is to compare the similarity between the information sub-matrix and the knowledge information. The specific steps include: encoding the knowledge information to obtain a knowledge information matrix, and calculating the similarity value between the information sub-matrix and the knowledge information matrix; then, comparing the similarity value with a preset similarity threshold. If the similarity threshold is greater than or equal to the similarity threshold, it is determined that the output result utilizes the first type of knowledge information.

[0078] The above similarity calculation algorithms also include but are not limited to cosine similarity algorithm or word2vec algorithm.

[0079] It should be pointed out that the first type of knowledge information refers to information in a vertical field, which can be information in a vertical field knowledge base or information similar thereto.

[0080] In summary, the above-mentioned embodiment converts inputs such as user requests into an N*M matrix and passes it into a large language model based on the Transformer architecture for calculation. An N*N weight matrix will be generated in the middle. The N*N weight of the input segment is retained in the K-times autoregressive predictions and the K-times average is taken to obtain the average weight of the input segment of length N in the entire output process. For the (N_1~N) content where the knowledge information is located, the average value of the sub-matrix at the corresponding position is taken to obtain the weight of the knowledge information. If it is less than the preset empirical threshold, it is considered that the knowledge is not effectively utilized.

[0081] In some embodiments, in combination Figure 4 In the method flow shown in FIG. 1 , step S130 inputs the model input matrix into the large language model to obtain an intermediate matrix and an output matrix, and encodes the output matrix to obtain an output answer, further comprising:

[0082] Convert each word element vector in the output matrix into a unit vector whose sum is 1; specifically, for a 1*M vector of K dimension, convert it into a unit vector through a softmax algorithm;

[0083] According to the unit vector of K dimension, the distribution entropy value of each unit vector is calculated. The calculation can be performed using the following calculation formula:

[0084] Calculate the arithmetic mean of the K distribution entropy values, and use the arithmetic mean as the confidence of the answer;

[0085] Compare the answer confidence value with the preset confidence threshold. If it is less than or equal to the confidence threshold, it means that the answer confidence is good, and the answer can be output to the user. If it is greater than the confidence threshold, it means that the surface confidence does not meet the requirements, and human experts can be introduced to modify it, and then the modified answer can be stored in the vertical field knowledge base.

[0086] This embodiment determines whether the confidence of the answer is high enough by the concentration degree of probability distribution of different characters in the neural network output token of the large language model.

[0087] In some embodiments, the method further comprises:

[0088] If it is determined that the output answer does not utilize the first type of knowledge information or the confidence value of the answer is greater than a confidence threshold, the output result is output to a human expert so that the human expert can provide modification suggestions.

[0089] According to this embodiment, combined with Figure 4 As shown in the figure, if the knowledge information in the "vertical domain knowledge base" is not effectively utilized, it is submitted to human experts for verification, and the human experts make modification suggestions and obtain the modified results; if the knowledge information is effectively utilized, it continues to determine whether the confidence of the generated results is high enough. If it is not high enough, it is submitted to human experts for verification, and modification suggestions are made and the modified results are obtained. The modified answers can be directly stored in the vertical domain knowledge base. In addition, the processing process and direct output results of the large language model can also be uploaded to the vertical domain knowledge base.

[0090] Specifically, according to the example in the above embodiment, the final output result may be: Your vehicle maintenance generally takes four hours, subject to the actual service progress.

[0091] In some embodiments, the method further includes: using knowledge information in a vertical domain knowledge base to supervise the training of the large language model.

[0092] Specifically, the knowledge information in the vertical domain knowledge base can also be directly injected into the large language model based on the Transformer architecture through supervised training. Optionally, if knowledge injection is performed in this way, there is no need to select knowledge through the semantic similarity algorithm and fill in the prompt word template. You only need to fill the user request into the system prompt word template and then pass it to the Tokenizer.

[0093] In some embodiments, the vertical field is the field of smart electric vehicles, and the first category of knowledge information is smart electric vehicle knowledge, including at least one of the following: new product development information, user vehicle service information, production line upgrade and transformation information, vehicle failure causes and solutions information, or user service standard operating procedures.

[0094] According to this embodiment, the vertical field can be the field of smart electric vehicles. Since company-level information cannot be included in the corpus for training the large language model due to commercial confidentiality or user information confidentiality, it can only be trained within the company or continuously enriched or injected during the actual use of the large language model. Specific knowledge information includes but is not limited to information on new product development of electric vehicles, user car services, production line upgrades and transformations, etc. Specific data examples include design technical details of electric vehicles, vehicle failure causes and solutions, user service standard question and answer SOP, etc.

[0095] Combination Figure 5 As shown, another embodiment of the present invention further protects a vertical field information processing device 500 based on a large language model, and the vertical field information processing device 500 based on a large language model includes:

[0096] A similarity calculation module 510 is adapted to receive a user request and determine the knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm;

[0097] A filling encoding module 520 is adapted to determine a prompt word template according to the user request and / or the knowledge information, fill the knowledge information into the prompt word template according to a preset format, and encode the filled prompt word template to obtain a model input matrix;

[0098] A model operation module 530, adapted to input the model input matrix into a large language model, obtain an intermediate matrix and an output matrix, and decode the output matrix to obtain an output answer;

[0099] A position determination module 540, adapted to determine an information sub-matrix corresponding to the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix;

[0100] The effect evaluation module 550 is adapted to determine whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

[0101] According to the vertical domain information processing device based on the large language model disclosed in this embodiment, on the one hand, the large language model is used to liberate the energy of data engineers and user service personnel and reduce the cost of information processing; on the other hand, by introducing modules such as effect evaluation, the ability of the large language model to accurately use the vertical domain knowledge base is improved, thereby solving the problem of inaccurate information processing results caused by the large language model not including vertical domain knowledge information in conventional training data, so that the general large language model can still achieve a high information processing accuracy rate when trained in the absence of vertical domain knowledge.

[0102] In some embodiments, assuming that the model input vector matrix is ​​an N*M matrix, and the output matrix is ​​a K*M matrix, where N is the number of word units in the prompt word template, K is the number of word units in the model output answer, M is the expression length of each word unit, and K, M, and N are positive integers greater than 1, then the model operation module 530 inputs the model input matrix into the large language model, and the intermediate matrix obtained includes:

[0103] Obtain K N*N weight matrices obtained by K operations, calculate the arithmetic mean of the K N*N weight matrices, obtain an arithmetic mean N*N matrix, and use the arithmetic mean N*N matrix as an intermediate matrix;

[0104] The location determination module 540 is also adapted to:

[0105] Determine the vector position of the knowledge information in the N*M matrix according to the position of the prompt word template filled with the knowledge information;

[0106] An information sub-matrix is ​​determined from the arithmetic mean N*N matrix according to the vector position.

[0107] In some embodiments, the effect evaluation module 550 is further adapted to:

[0108] Calculating the arithmetic mean of each value in the information submatrix, and determining whether the arithmetic mean is greater than a preset empirical threshold, and if it is not less than the empirical threshold, determining that the output result utilizes the first type of knowledge information; or,

[0109] Encoding the knowledge information to obtain a knowledge information matrix, and calculating a similarity value between the information sub-matrix and the knowledge information matrix;

[0110] The similarity value is compared with a preset similarity threshold, and if the similarity threshold is greater than or equal to the similarity threshold, it is determined that the output result uses the first type of knowledge information.

[0111] In some embodiments, the model operation module 530 is further adapted to:

[0112] Convert each word element vector in the output matrix into a unit vector whose sum is 1;

[0113] Calculate the distribution entropy value of each unit vector;

[0114] Calculate the arithmetic mean of the distribution entropy values, and use the arithmetic mean as the answer confidence;

[0115] The answer confidence value is compared with a preset confidence threshold, and the answer is output if the confidence value is less than or equal to the confidence threshold.

[0116] In some embodiments, the apparatus 500 is further adapted to:

[0117] If it is determined that the output answer does not utilize the first type of knowledge information or the confidence value of the answer is greater than a confidence threshold, the output result is output to a human expert so that the human expert can provide modification suggestions.

[0118] In some embodiments, the apparatus 500 is further adapted to:

[0119] The large language model is trained using knowledge information in a vertical domain knowledge base.

[0120] In some embodiments, the vertical field is the field of smart electric vehicles, and the first category of knowledge information is smart electric vehicle knowledge, including at least one of the following: new product development information, user vehicle service information, production line upgrade and transformation information, vehicle failure causes and solutions information, or user service standard operating procedures.

[0121] It should be noted that the specific implementation of the above-mentioned device embodiments can be carried out with reference to the specific implementation of the above-mentioned corresponding method embodiments, which will not be repeated here.

[0122] It should be noted that:

[0123] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general-purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for the purpose of disclosing the best mode of the present invention.

[0124] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0125] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0126] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0127] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0128] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a vertical field information processing device based on a large language model according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0129] An embodiment of the present invention provides a non-volatile computer storage medium, which stores at least one executable instruction, and the computer executable instruction can execute the vertical field information processing method based on a large language model in any of the above method embodiments.

[0130] Figure 6 A schematic diagram of the structure of an embodiment of a computing device of the present invention is shown. The specific embodiment of the present invention does not limit the specific structure of the computing device.

[0131] like Figure 6 As shown, the computing device may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0132] The processor 602, the communication interface 604, and the memory 606 communicate with each other via a communication bus 608. The communication interface 604 is used to communicate with other devices such as a client or other server network elements. The processor 602 is used to execute a program 610, which can specifically execute the relevant steps in the above-mentioned vertical domain information processing method embodiment based on a large language model for a computing device.

[0133] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0134] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0135] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0136] The program 610 can be specifically used to enable the processor 602 to perform operations corresponding to the above-mentioned vertical field information processing method embodiment based on a large language model.

[0137] Further, combined with Figure 7As shown, the large language model of the above embodiment of the present invention can be implemented by a service agent, and the answer confidence calculation and effect evaluation judgment are implemented by an effect evaluation agent. The effect evaluation agent and the service agent are respectively set on computing device 1 and computing device 2, and the vertical domain knowledge base is set on storage device 1, and the above devices are connected through communication interfaces and protocols.

[0138] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present invention can be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims

1. A vertical domain information processing method based on a large language model, characterized in that: The method comprises: Receive a user request, and determine the knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm; Determine a prompt word template according to the user request and / or the knowledge information, fill the knowledge information into the prompt word template according to a preset format, and encode the filled prompt word template to obtain a model input matrix; Inputting the model input matrix into a large language model to obtain an intermediate matrix and an output matrix, and decoding the output matrix to obtain an output answer; Determining, according to the position of the knowledge information in the model input matrix, an information submatrix corresponding to the knowledge information in the intermediate matrix; It is determined whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

2. The method according to claim 1, characterized in that Assuming that the model input vector matrix is ​​an N*M matrix, and the output matrix is ​​a K*M matrix, where N is the number of word units in the prompt word template, K is the number of word units in the model output answer, and M is the expression length of each word unit, the model input matrix is ​​input into the large language model, and the intermediate matrix obtained includes: Obtain K N*N weight matrices obtained by K operations, calculate the arithmetic mean of the K N*N weight matrices, obtain an arithmetic mean N*N matrix, and use the arithmetic mean N*N matrix as an intermediate matrix; Determining the information submatrix of the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix includes: Determine the vector position of the knowledge information in the N*M matrix according to the position where the knowledge information is filled into the prompt word template; An information sub-matrix is ​​determined from the arithmetic mean N*N matrix according to the vector position.

3. The method according to claim 1 or 2, characterized in that: Judging whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix includes: Calculating the arithmetic mean of each value in the information submatrix, and determining whether the arithmetic mean is greater than a preset empirical threshold, and if it is not less than the empirical threshold, determining that the output result utilizes the first type of knowledge information; or, Encoding the knowledge information to obtain a knowledge information matrix, and calculating a similarity value between the information sub-matrix and the knowledge information matrix; The similarity value is compared with a preset similarity threshold, and if the similarity threshold is greater than or equal to the similarity threshold, it is determined that the output result uses the first type of knowledge information.

4. The method according to claim 1, characterized in that: Inputting the model input matrix into the large language model to obtain an intermediate matrix and an output matrix, and encoding the output matrix to obtain an output answer includes: Convert each word element vector in the output matrix into a unit vector whose sum is 1; Calculate the distribution entropy value of each unit vector; Calculate the arithmetic mean of the distribution entropy values, and use the arithmetic mean as the answer confidence; The answer confidence value is compared with a preset confidence threshold, and the answer is output if the confidence value is less than or equal to the confidence threshold.

5. The method according to claim 4, characterized in that The method further comprises: If it is determined that the output answer does not utilize the first type of knowledge information or the confidence value of the answer is greater than a confidence threshold, the output result is output to a human expert so that the human expert can provide modification suggestions.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: The large language model is trained using knowledge information in a vertical domain knowledge base.

7. The method according to claim 1 or 2, characterized in that: The vertical field is the field of smart electric vehicles, and the first category of knowledge information is smart electric vehicle knowledge, including at least one of the following: new product research and development information, user vehicle service information, production line upgrade and transformation information, vehicle failure causes and solution information, or user service standard operating procedures.

8. A vertical domain information processing device based on a large language model, characterized in that: The device comprises: A similarity calculation module, adapted to receive a user request and determine the knowledge information closest to the user request from a vertical domain knowledge base according to the user request through a semantic similarity algorithm; A filling encoding module, adapted to determine a prompt word template according to the user request and / or the knowledge information, fill the knowledge information into the prompt word template according to a preset format, and encode the filled prompt word template to obtain a model input matrix; A model operation module, adapted to input the model input matrix into a large language model, obtain an intermediate matrix and an output matrix, and decode the output matrix to obtain an output answer; a position determination module, adapted to determine an information submatrix corresponding to the knowledge information in the intermediate matrix according to the position of the knowledge information in the model input matrix; The effect evaluation module is adapted to determine whether the output answer utilizes the first type of knowledge information in the vertical domain knowledge base according to the information sub-matrix.

9. A computing device, characterized in that The invention comprises a processor and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the vertical domain information processing method based on a large language model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the vertical domain information processing method based on a large language model according to any one of claims 1 to 7.

Images