Power grid marketing data labeling method based on confusion-driven large language model

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a power grid marketing data labeling method, platform and equipment based on a confusion-driven large language model. The method comprises the following steps: S1, collecting related data according to specific requirements in the field of power grid marketing, and preprocessing the data to form a required sample data set; s2, questions related to power grid marketing business are collected, and the questions are answered by the large language model to construct an initial question-answer pair; s3, confusion detection is carried out on the initial question and answer pairs in combination with the sample data set, and the initial question and answer pairs with accuracy risks are screened out; and S4, performing manual examination and correction on the initial question and answer pairs with the accuracy risk by a technician to form annotation data, collecting a large amount of data subjected to manual examination and annotation, and forming a knowledge base for performing re-supervision training on a commercial large language model. According to the method, the problem that the data labeling task load of a worker is too large due to model illusion of a large power grid marketing model is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a power grid marketing data labeling method based on a perplexity-driven large language model and an interactive platform and equipment for corresponding power grid marketing services. Background Art

[0002] With the rapid development and widespread application of large language models (LLMs), they have shown significant potential in processing rules and policy documents in the power grid industry, such as power supply business rules. Traditionally, large model training in the field of power grid marketing relies on manually labeled and designed training data, which is not only time-consuming but also costly. Large models can assist in the identification and standardization of sample data sets. However, when processing documents containing a large amount of data information, large models often produce some hallucination problems on these numbers during the data generation process, which may lead to misunderstanding or incorrect application of rules and technical specifications.

[0003] Although there is no specific solution to the problem of hallucinations in large language models when generating data, some methods have been proposed to deal with the problem of hallucinations in large language models in regular conversations. The main solution is divided into two stages: model training and application, aiming to effectively alleviate model hallucinations. In the training stage, the solution focuses on building high-quality sample data sets, optimizing the model structure to better utilize source information, and introducing controllable text generation and reinforcement learning techniques. In the application stage, relevant information is recalled by connecting to an external knowledge base, and the output of the model is detected, identified, and corrected for any hallucinations, thereby reducing the occurrence of hallucinations in the text generation process.

[0004] Regarding the mitigation of hallucinations during the application phase, the research of technicians has proposed the following various solutions for post-editing and correcting the hallucinations produced by the model:

[0005] The technical solution disclosed in the Chinese invention patent application publication number CN117875327A uses a large language model to generate content based on key information and uses a named entity recognition model to detect entities in the text. False sentences are located by calculating the entity difference between the entities in the generated content and the key information. The system uses a local knowledge base or other external real information base to further verify these false sentences and determine the real false sentences and the corresponding real information. Finally, the system combines the generated content, false sentences and real information to construct prompt information for hallucination repair based on the large language model. This method can not only effectively locate false information in the generated content, but also provide a repair mechanism to enhance the authenticity and reliability of the generated content. Although the solution can effectively identify predefined entity types using named entity recognition (NER) technology, when faced with complex documents such as "Order No. 14 of Power Supply Business Rules" that contain a large amount of professional terms and digital information, it may not be deep enough in understanding the contextual logical relationships and non-standard entities in the document. Especially in professional fields, NER systems often require a lot of customized training to ensure the accuracy of recognition. In addition, NER technology may face efficiency issues when processing large-scale data, especially when performing operations such as entity difference calculation and false information location, the process may be particularly time-consuming.

[0006] China Invention Patent Application Publication No. CN117931983A discloses a method and system for generating accurate answers using a large model in the field of natural language processing. The method mainly includes the following steps: preprocessing texts from different source fields, importing the preprocessed texts into Elasticsearch, and managing them through a predefined ES index mapping structure; then, building a custom retrieval tool based on LangChain to return relevant documents, and embedding the processed texts into pre-designed prompt frameworks, and the large model generates the final answers to user questions based on the external information contained in these frameworks. The scheme demonstrates a powerful ability to generate accurate answers, but there are still some challenges in its implementation. First, the system relies on a complex data processing and retrieval system, which not only requires a large amount of storage and computing resources, but also may lead to performance bottlenecks when dealing with technical issues involving multiple voltage parameters and specifications. Especially in the case of large data volumes or complex queries, the multi-step processing process (from data preprocessing, indexing, retrieval to answer generation) may cause system response time delays. In addition, the system's performance and response speed greatly depend on the structure of the data index and the optimization of the retrieval strategy. If the indexing or retrieval strategy is not designed properly, it may affect the accurate retrieval and efficient processing of complex voltage types and allowable deviation data.

[0007] Chinese invention patent application publication number CN117540811A discloses a system and method applied to the field of large language model technology, aiming to solve the hallucination problem in large language models. Its workflow is: first, the question information and its related data (including the first search terms and search results) are input into the preprocessing large model, and a second more accurate search term is generated based on this information. Then a second search is performed in the database to obtain new search results. Finally, the system uses all this information (including the search results of two rounds and traceable content) for the correction module to correct and optimize the traceable content initially generated. The scheme adopts a multi-step systematic process to improve the accuracy of the model output, which involves multiple rounds of iterative search and correction. For example, when dealing with the aforementioned specific problem, each round of iteration requires the retrieval and analysis of a large amount of document data containing voltage levels and deviation parameters. Although this method aims to improve the accuracy of information through continuous correction, multiple iterations may not only increase the complexity of the system, but also lead to longer running time. More importantly, there is also a risk of mistakenly changing the originally accurate information during each correction process, which may lead to the accumulation of new errors. Especially when the amount of data is large, even a small proportion of miscorrection may affect the accuracy of the final result, causing the large model to introduce new errors in the process of trying to solve the hallucination problem. Therefore, this method may face challenges in efficiency and potential error control while ensuring output quality. Summary of the invention

[0008] In order to solve the model hallucination problem existing in the large model used in the power grid marketing business, the present invention provides a power grid marketing data labeling method based on perplexity-driven large language model and its corresponding interactive platform and equipment for power grid marketing business.

[0009] The technical solution provided by the present invention is:

[0010] A method for labeling power grid marketing data based on a perplexity-driven large language model, comprising the following steps:

[0011] S1: According to the specific needs in the field of power grid marketing, relevant data are collected and preprocessed to form the required sample data set.

[0012] S2: Collect questions related to power grid marketing business, input them into any commercial large language model, and construct a series of initial question-answer pairs based on the content generated by the large language model.

[0013] S3: Perform perplexity detection on the initial question-answer pairs constructed by the large language model based on the sample dataset, and screen out initial question-answer pairs with accuracy risks.

[0014] S4: Technical personnel will manually review and correct the initial question-answer pairs that have accuracy risks to form labeled data. A large amount of manually reviewed and labeled data will be collected to form a knowledge base for re-supervised training of large commercial language models.

[0015] As a further improvement of the present invention, in step S1, the relevant data collected include power supply business rules, industry regulations, pricing strategy information, business expansion registration, operating specifications, market reports, and collection information.

[0016] During the data preprocessing process, the numerical data is cleaned and normalized, and the text data is extracted from the collected relevant data, and then formatted and vectorized.

[0017] As a further improvement of the present invention, in the data cleaning process, statistical regularization technology is first used to identify and correct outliers in the sample data set, and then isolation forests, autoencoders or other anomaly detection algorithms are used to identify and correct errors in the data, thereby removing irrelevant, duplicate or missing data.

[0018] As a further improvement of the present invention, the text data in the sample data set is preprocessed and vectorized through the LangChain framework.

[0019] As a further improvement of the present invention, the large language models used include ChatGLM and GPT-3; wherein, the purchased large language model service is applied by calling an API interface.

[0020] As a further improvement of the present invention, in step S3, the perplexity detection method of the QA pair is as follows:

[0021] S31: The probability p of each question in the QA pair belonging to the i-th category after being transformed by the softmax function is calculated by the following formula: i :

[0022]

[0023] In the above formula, z j represents the jth answer vector; n represents the total number of categories; Yes i The index score of It is the sum of the index scores of all categories, ensuring that the sum of all probabilities is 1.

[0024] S32: Calculate the entropy H(p) of the probability distribution of the current question pair by the following formula i ):

[0025]

[0026] S33: According to the entropy H(p i ) and the preset threshold H0 make the following judgment:

[0027] (1) When H(p i )>H0, it means that the problem has accuracy risk;

[0028] (2) When H(p i )≤H0, it means that the problem has no accuracy risk.

[0029] As a further improvement of the present invention, in step S4, for the knowledge base obtained after manual review, a decision tree or a random forest is used to classify and regress the content items in the knowledge base to verify its accuracy and optimize the information structure.

[0030] The present invention also includes a power grid marketing business interactive platform, which is used to adopt the power grid marketing data labeling method based on the perplexity-driven large language model as described above to overcome the model hallucination problem of the large language model used in the power grid marketing business. The power grid marketing business interactive platform includes: a model calling unit, a knowledge base, a local question-and-answer unit, a perplexity calculation unit, and an interactive unit.

[0031] Among them, the model calling unit is used to analyze the questions raised by users in marketing business interactions, and then use the preset API interface to call the question and answer service of the large language model to obtain answers to the questions.

[0032] The knowledge base is constructed using the aforementioned power grid marketing data annotation method based on the perplexity-driven large language model, and is updated in real time during dynamic interaction with users.

[0033] The local question-answering unit runs a local model trained with the knowledge base as supervisory data. The local model is used to generate corresponding answers based on input power grid marketing-related questions.

[0034] The perplexity calculation unit is used to calculate the perplexity and entropy of probability distribution of each QA pair produced by the large language model in combination with the content of the knowledge base, so as to evaluate whether there is an accuracy risk in the answer to the question;

[0035] The interaction unit is used to obtain the evaluation results of the perplexity calculation unit and make the following decisions: when the answer generated by the large language model does not have accuracy risks, it will be used as the answer to the question raised by the user and returned to the user; when the answer generated by the large language model has accuracy risks, the local model will be used to generate the answer to the corresponding question and then returned to the user.

[0036] As a further improvement of the present invention, the API interface adopted by the model calling unit supports encrypted data transmission and identity authentication functions to ensure the security of data during transmission and to manage user access rights.

[0037] The present invention also includes a power grid marketing business interactive device, which includes a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, it creates the power grid marketing business interactive platform as described above; realizes user interaction based on a large language model, and alleviates the model illusion existing in the large language model.

[0038] The technical solution provided by the present invention has the following beneficial effects:

[0039] The present invention provides another power grid marketing data labeling method based on perplexity-driven big language model. The method first collects a large amount of sample data related to power grid marketing, and generates a large number of question-answer pairs through commercial big language. Then, the collected sample data is combined to confirm the accuracy of the question-answer pairs generated by the big model through perplexity analysis method. Finally, the question-answer pairs with low accuracy are manually reviewed and modified. Finally, a large amount of manually corrected labeled data is used as a knowledge base for supervised learning of the big model to overcome the model hallucination problem of the big model.

[0040] The solution proposed by the present invention helps to solve the hallucination problem that occurs when the existing large language model generates data in the field of power grid marketing. This method of synthesizing data with a large model combined with perplexity detection and knowledge retrieval enhancement can enhance the performance of the large model when processing professional data of power grid marketing, and also help to optimize the diversity and coverage of data, thereby reducing the workload of business managers. In addition, the technical solution of the present invention is particularly suitable for processing highly variable and highly professional professional data of power grid marketing, such as voltage levels and power supply rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of the steps of the power grid marketing data labeling method based on the perplexity-driven large language model provided in Example 1 of the present invention.

[0042] Figure 2 Schematic diagram of the knowledge base retrieval and information verification process in Example 1 of the present invention.

[0043] Figure 3 This is a system topology diagram of the power grid marketing business interactive platform provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] Example 1

[0046] This embodiment provides a method for annotating power grid marketing data based on a perplexity-driven large language model. The method can use a large language model to automatically generate high-quality training data, then perform perplexity analysis on the generated data, and correct the erroneous data. The perplexity correction mechanism significantly reduces the illusion problem of generated data, thereby ensuring the accuracy of the data. This method is used to reduce the dependence of standard tasks of power grid marketing data on manual intervention and save human resources. It sets a new benchmark for the level of automation and intelligence of data collection in the field of power grid marketing. This solution helps to enhance the performance of large models when processing professional data for power grid marketing.

[0047] Specifically, Figure 1 As shown, the power grid marketing data labeling method based on perplexity-driven large language model provided in this embodiment includes the following steps:

[0048] S1: According to the specific needs in the field of power grid marketing, relevant data are collected and preprocessed to form the required sample data set.

[0049] In this embodiment, the type of data collected needs to be determined according to the specific application scenario. For example, in typical scenarios such as power supply business rule consultation and intelligent customer service, it is necessary to first conduct a demand analysis to analyze the key issues and required data types in each scenario. Subsequently, relevant data is collected from various different data sources, such as internal enterprise databases, public information release platforms, or real-time data collected directly from field equipment. This step is to ensure the comprehensiveness and diversity of the data, laying the foundation for subsequent data processing and model training. In the scheme of this embodiment, the relevant data collected includes power supply business rules, industry regulations, pricing strategy information, business expansion registration, operating specifications, market reports, collection information, etc.

[0050] During the data preprocessing process, this embodiment divides the collected data into two categories: numerical data and text data, and performs data cleaning and normalization on the numerical data. In the data cleaning process of numerical data, statistical regularization technology is first used to identify and correct outliers in the sample data set. This method identifies outliers by calculating the standard deviation of data points from the mean, and then adjusts these data points to the standard normal distribution range to ensure that all data points are within the predetermined threshold. These steps ensure the quality and consistency of the input data, which is crucial to improving the efficiency and accuracy of model training. Then, isolation forests, autoencoders or other anomaly detection algorithms are used to identify and correct errors in the data, thereby removing irrelevant, duplicated or missing data.

[0051] Among them, the data cleaning operation includes: missing value processing, outlier processing, and duplicate data removal. Missing value processing refers to filling or processing these missing values ​​through appropriate strategies when there are null values ​​or missing data in the data set to reduce the impact on subsequent analysis and model training. This embodiment uses an interpolation algorithm to fill the missing values. Outlier processing refers to identifying and processing observations that are significantly different from other data points. Outliers may be data entry errors, measurement errors, or actual extreme phenomena. The outlier processing methods used in this embodiment include z-score, IQR method, and model-based methods such as isolation forests. Duplicate data removal is to avoid information redundancy in analysis and ensure the uniqueness of the data set by identifying the same records in the data set. Duplicate data is usually detected by finding exactly the same rows or partially identical features.

[0052] For text data, this embodiment first extracts the text data from the collected relevant data, and then formats and vectorizes it. Before text analysis, the original text data needs to be processed to remove noise, unify the format, and extract useful information. The processing steps include removing punctuation, converting to lowercase, removing stop words, and stemming. Next, this embodiment completes the preprocessing and vectorization of the text data in the sample data set through the LangChain framework.

[0053] S2: Collect questions related to power grid marketing business, input them into any commercial large language model, and construct a series of initial question-answer (OA) pairs based on the content generated by the large language model.

[0054] In this embodiment, the large language model used includes ChatGLM, GPT-3 or other large language models that can handle power grid marketing business. Among them, the solution implemented in this embodiment can choose to purchase the cloud service of the large language model, and then implement the application by calling the API interface.

[0055] For example, suppose the collected question 1 is:

[0056] “What types of rated voltages are supplied by power supply companies?”

[0057] At this point, the large language model gives the corresponding answer:

[0058] "(i) In the case of low voltage power supply: single phase is 220 volts, three phase three wire is 380 volts, three phase four wire is 380 / 220 volts. (ii) In the case of high voltage power supply: 10 (6, 20), 35, 110 (66), 220 (330) kilovolts.

[0059] When the voltage level required by the user is not within the above range, the user shall take voltage transformation measures to solve the problem. When the voltage level required by the user is above 110 kV, the power receiving device shall be designed as a terminal substation. "

[0060] Assume that question 2 is: "What is the allowable deviation of the power supply voltage supplied by the power supply enterprise to the user's receiving end?"

[0061] At this point, the answer given by the large language model is:

[0062] "(i) For power supply with voltage above 35 kV, the sum of the absolute values ​​of positive and negative voltage deviations shall not exceed 10% of the rated value. (ii) For three-phase power supply with voltage below 10 (6, 20) kV, it shall be ±7% of the rated value; (iii) For single-phase power supply with voltage of 220 V, it shall be +7% or -10% of the rated value.

[0063] Under abnormal conditions of the power system, the maximum allowable voltage deviation at the user's receiving end shall not exceed ±10% of the rated value. If the user's power factor does not meet the requirements of Article 45 of these rules, the voltage deviation at the receiving end shall not be subject to this restriction. "

[0064] S3: Perform perplexity detection on the initial question-answer pairs constructed by the large language model based on the sample dataset, and screen out initial question-answer pairs with accuracy risks.

[0065] The perplexity detection method for QA pairs is as follows:

[0066] S31: The probability p of each QA question belonging to the i-th category after being transformed by the softmax function is calculated by the following formula: i :

[0067]

[0068] In the above formula, z j represents the jth answer vector; n represents the total number of categories; Yes i The index score of It is the sum of the index scores of all categories, ensuring that the sum of all probabilities is 1.

[0069] S32: The uncertainty of probability can be measured by the entropy of the probability distribution, which is a measure of randomness. For the probability distribution p of the current problem i , the entropy H(p i ):

[0070]

[0071] S33: According to the entropy H(p i ) and the preset threshold H0 make the following judgment:

[0072] (1) When H(p i )>H0, it means that the problem has accuracy risk;

[0073] (2) When H(p i )≤H0, it means that the problem has no accuracy risk.

[0074] S4: Technical personnel manually review and correct QA pairs with accuracy risks to form labeled data. A large amount of manually reviewed and labeled data is collected to form a knowledge base for re-supervised training of large commercial language models.

[0075] In the solution of this embodiment, based on the results of the perplexity evaluation, the part of the model that is not confident will be supported or corrected by retrieving conclusive information from a knowledge base designed specifically for the field of power grid marketing. The knowledge base consists of precise information from the laws and regulations, operating specifications, historical electricity consumption data, and the latest market analysis reports of the power grid industry. In addition, for the knowledge base obtained after manual review, decision trees or random forests are used to classify and regress the content items in the knowledge base to verify its accuracy and optimize the information structure.

[0076] In the correction phase of the question-answer pairs generated by the large model, such as Figure 2As shown, for question 1 ("What types of rated voltages are provided by power supply companies?" "(i) In the case of low-voltage power supply: single-phase is 220 volts, three-phase three-wire is 380 volts, and three-phase four-wire is 380 / 220 volts. (ii) In the case of high-voltage power supply: 10 (6, 20), 35, 110 (66), 220 (330) kilovolts. When the voltage level required by the user is not within the above range, the user should take voltage transformation measures to solve it. When the voltage level required by the user is above 110 kilovolts, the power receiving device should be designed as a terminal substation.") The model's predictions are more confusing for the digital tokens 220, 380, 10 (6, 20), 35, 110 (66), and 220 (330). Management personnel need to confirm or correct them based on their own patent knowledge, or retrieve corresponding information from relevant knowledge bases.

[0077] Based on the above method, this embodiment can answer all high-frequency questions in the power grid marketing business, and mark the unclear answers in the question-and-answer results, and finally manually modify these wrong answers. The modified marked data is recorded as the part of the large language model where the model hallucination exists. This embodiment can use these data in the knowledge base as a verification data set to train, test and verify the offline local model to ensure the accuracy and consistency of the model output, and then adjust the model or data set according to the test results to meet the project requirements and business goals.

[0078] Example 2

[0079] On the basis of the scheme in Example 1, this embodiment also provides a power grid marketing business interactive platform, which is used to adopt the power grid marketing data annotation method based on the perplexity-driven large language model as in Example 1 to overcome the model hallucination problem of the large language model used in the power grid marketing business. The operating mechanism of the power grid marketing business interactive platform provided in this embodiment is: for each question raised by the user, first call the commercial large language model to answer, then perform perplexity analysis on the answer result, and return it directly to the user when the accuracy of the answer result meets the requirements. If the accuracy of the answer result is insufficient, the question and answer pair QA is annotated, and the annotated results are stored in a knowledge base after manual review and correction, and then use the knowledge base to train an offline local model so that the discrete local model has the ability to correctly answer the corresponding questions, and finally return the answer of the local model to the user. In this embodiment, the offline local model only needs to be trained for the problem that the large language model has hallucinations, so that the workload of manually collecting corresponding sample data can be greatly reduced. In the actual application process, as the amount of information collected by the knowledge base gradually increases, the trained local model can gradually complement the large language model to achieve accurate answers to all questions raised by the user.

[0080] Specifically, Figure 3 As shown, the power grid marketing business interactive platform provided by this embodiment includes: a model calling unit, a knowledge base, a local question and answer unit, a perplexity calculation unit and an interactive unit.

[0081] The model calling unit is used to analyze the questions raised by the user in the marketing business interaction, and then use the preset API interface to call the question-answering service of the large language model to obtain the answer to the question. In this embodiment, the API interface used supports encrypted data transmission and identity authentication functions, as well as ensuring the security of data during transmission and realizing the management of user access rights.

[0082] The perplexity calculation unit is used to calculate the perplexity and entropy of the probability distribution of each QA pair produced by the large language model in combination with the content of the knowledge base, and then evaluate whether the answer to the question has an accuracy risk. The knowledge base is constructed using the power grid marketing data annotation method based on perplexity-driven large language model as in Example 1, and is updated in real time during the dynamic interaction with the user.

[0083] The local question-answering unit runs a local model trained with the knowledge base as supervision data, and is used to generate corresponding answers based on the input power grid marketing related questions. When training the local model, the configuration of the network model can be dynamically adjusted according to the characteristics of the input data, including but not limited to the number of layers, number of hidden units and number of attention heads of the model, etc.

[0084] The interaction unit is used to obtain the evaluation results of the perplexity calculation unit and make the following decisions: when the answer generated by the large language model does not have accuracy risks, it will be used as the answer to the question raised by the user and returned to the user; when the answer generated by the large language model has accuracy risks, the local model will be used to generate the answer to the corresponding question and then returned to the user.

[0085] In this embodiment, the interactive unit uses a local question-answering unit as the fault-tolerant mechanism of the entire system. The local model can automatically trigger the correction process when the prediction of the large language model deviates, thereby enhancing the robustness of the system and the trust of users. In addition, the local model can also be used as the redundancy of the large language model to overcome the disadvantage that the real-time performance of the large language model application process will be affected by the network communication quality.

[0086] Example 3

[0087] Based on the solutions in Examples 1 and 2, this embodiment further provides a power grid marketing business interactive device, which includes a memory, a processor, and a computer program stored in the memory and running in the processor. When the processor executes the computer program, a power grid marketing business interactive platform as in Example 2 is created; user interaction based on a large language model is realized, and the model illusion existing in the large language model is alleviated.

[0088] The power grid marketing business interactive device provided in this embodiment is essentially a computer device for implementing the solutions in embodiments 1 and 2. The computer device in this embodiment can also be an intelligent terminal capable of executing a program, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc.

[0089] The computer device indicated in this embodiment includes at least but is not limited to: a memory and a processor that can be connected to each other through a system bus. Among them, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0090] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run program codes stored in a memory or process data.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for labeling power grid marketing data based on perplexity-driven large language model, characterized in that: It includes the following steps: S1: According to the specific needs of the power grid marketing field, collect relevant data and perform data preprocessing to form the required sample data set; S2: Collect questions related to power grid marketing business, input them into any commercial large language model, and construct a series of initial question-answer pairs based on the content generated by the large language model; S3: performing perplexity detection on the initial question-answer pairs constructed by the large language model in combination with the sample data set, and screening out initial question-answer pairs with accuracy risks; S4: Technical personnel will manually review and correct the initial question-answer pairs that have accuracy risks to form labeled data. A large amount of manually reviewed and labeled data will be collected to form a knowledge base for re-supervised training of large commercial language models.

2. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 1, characterized in that: In step S1, the relevant data collected include power supply business rules, industry regulations, pricing strategy information, business expansion registration, operation specifications, market reports, and collection information; During the data preprocessing process, the numerical data is cleaned and normalized, and the text data is extracted from the collected relevant data, and then formatted and vectorized.

3. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 2, characterized in that: In the data cleaning process, statistical regularization techniques are first used to identify and correct outliers in the sample data set, and then isolation forests, autoencoders or other anomaly detection algorithms are used to identify and correct errors in the data, thereby removing irrelevant, duplicate or missing data.

4. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 2, characterized in that: The text data in the sample dataset is preprocessed and vectorized through the LangChain framework.

5. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 1, characterized in that: The large language models used include ChatGLM and GPT-3. The purchased large language model service is applied by calling the API interface.

6. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 5, characterized in that: In step S3, the perplexity detection method of the QA pair is as follows: S31: The probability p of each question in the QA pair belonging to the i-th category after being transformed by the softmax function is calculated by the following formula: i : In the above formula, z j represents the jth answer vector; n represents the total number of categories; Yes i The index score of It is to sum the index scores of all categories, ensuring that the sum of all probabilities is 1; S32: Calculate the entropy H(p) of the probability distribution of the current question pair by the following formula i ): S33: According to the entropy H(p i ) and the preset threshold H0 make the following judgment: (1) When H(p i )>H0, it means that the problem has accuracy risk; (2) When H(p i )≤H0, it means that the problem has no accuracy risk.

7. The method for labeling power grid marketing data based on perplexity-driven large language model according to claim 1, characterized in that: In step S4, for the knowledge base obtained after manual review, a decision tree or random forest is used to classify and regress the content items in the knowledge base to verify its accuracy and optimize the information structure.

8. A power grid marketing business interactive platform, which is used to adopt the power grid marketing data labeling method based on perplexity-driven large language model as described in any one of claims 1 to 7 to overcome the model hallucination problem of the large language model used in the power grid marketing business; the power grid marketing business interactive platform comprises: The model calling unit is used to analyze the questions raised by users in marketing business interactions, and then use the preset API interface to call the question-answering service of the large language model to obtain answers to the questions; A knowledge base, which is constructed by using the power grid marketing data annotation method based on the perplexity-driven large language model as described in any one of claims 1 to 7, and is updated in real time during the dynamic interaction with the user; A local question-answering unit, in which a local model trained by using the knowledge base as supervisory data is run, and the local model is used to generate corresponding answers according to input power grid marketing related questions; A perplexity calculation unit, which is used to calculate the perplexity and entropy of probability distribution of each QA pair produced by the large language model in combination with the content of the knowledge base, and then evaluate whether there is an accuracy risk for the question; An interactive unit is used to obtain the evaluation result of the perplexity calculation unit and make the following decisions: when the answer generated by the large language model does not have an accuracy risk, it is used as the answer to the question raised by the user and returned to the user; when the answer generated by the large language model has an accuracy risk, the local model is used to generate the answer to the corresponding question, and then it is returned to the user.

9. The power grid marketing business interactive platform according to claim 8, characterized in that: The API interface used by the model calling unit supports encrypted data transmission and identity authentication functions to ensure the security of data during transmission and to manage user access rights.

10. A power grid marketing business interactive device, comprising a memory, a processor, and a computer program stored in the memory and running in the processor, characterized in that: When the processor executes the computer program, it creates the power grid marketing business interactive platform as described in claim 8 or 9; realizes user interaction based on the large language model, and alleviates the model illusion existing in the large language model.

Images