Knowledge question and answer model based on differential privacy and large model thinking chain

Abstract

The invention discloses a knowledge question and answer model based on differential privacy and a large model thinking chain in the technical field of artificial intelligence, and the model comprises an interaction module which is used for collecting and grading user privacy preference information; the data processing module is used for constructing a database; the disturbance module is used for adding disturbance quantity to the database; the problem processing module is used for preprocessing input problems; the intention recognition module is used for recognizing key entities and performing judgment and classification; the answer generation module is used for answering based on a GPT large language model; and the storage module is used for storing the interaction record. According to the method, the structure is perfect, the user privacy preferences in the database are queried in advance, the database is graded according to the privacy preferences, the disturbance quantity of differential privacy to the database is adjusted according to grading, and the accuracy of the knowledge question-answer model is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically to a knowledge question-answering model based on differential privacy and large model thinking chain. Background Art

[0002] With the rapid development of information technology, the advent of the big data era has made people face an unprecedented information explosion. In professional fields, various types of knowledge continue to emerge, and the acquisition, management and application of this knowledge have become increasingly complex. An efficient and intelligent professional knowledge question-and-answer system has become an urgent need to solve the problem of knowledge management in professional fields.

[0003] With the rise of large language models, the field of intelligent question answering has undergone a major transformation. Large language models have shown great potential in tasks such as question answering systems. Large language models can enable computers to better understand and generate natural language, which helps enhance the ability of question answering systems to understand complex questions raised by users. Thinking Chain is a fine-tuning technology for large language models. The main idea is to show a small number of examples to the large language model and explain the reasoning process in the examples. The large language model will also display the reasoning process when answering prompts, leading to more accurate results.

[0004] However, large language models usually face some security and privacy challenges. Since the databases they train and call are extremely large and contain a large amount of sensitive information, once they are attacked or leaked, there is a risk of leaking user privacy, which will not only endanger the privacy and security of users, but also affect the reliability and credibility of large language models. Differential privacy is a powerful privacy protection technology that can effectively protect the privacy of large language models during training and inference. The core idea is to add random noise during model training or inference so that the output of the model is less sensitive to any single training sample or input sample, thereby achieving the purpose of privacy protection.

[0005] At present, in the process of using differential privacy technology to protect the privacy of knowledge question answering models, since differential privacy requires adding random noise to the database, it will make the large language model training convergence low and the data availability poor, thus reducing the accuracy of knowledge question answering. Therefore, it is necessary to propose a knowledge question answering model based on differential privacy and large model thinking chain. Summary of the invention

[0006] In order to solve the problem that the above-mentioned differential privacy adds random noise, resulting in a decrease in the accuracy of the model knowledge question and answer model, the purpose of the present invention is to provide a knowledge question and answer model based on differential privacy and a large model thinking chain. By pre-querying the user privacy preferences in the database, grading the database according to the privacy preferences, and adjusting the amount of disturbance of the database by differential privacy according to the grading, the accuracy of the knowledge question and answer model can be effectively improved.

[0007] In order to achieve the above purpose, the technical solution of the present invention is as follows: a knowledge question-answering model based on differential privacy and large model thinking chain, including:

[0008] The interactive module is used to collect the privacy preference information of users before uploading data through the front-end interface, and classify the data uploaded by users into several privacy protection levels;

[0009] The data processing module is used to collect documents, extract document contents, build a database, and divide the database into several levels according to the privacy protection level;

[0010] The perturbation module is used to add perturbations to the database. Each level of database corresponds to a different perturbation amount. The higher the database privacy protection level, the greater the perturbation amount, and the perturbation amount parameter satisfies a linear relationship with the database level.

[0011] The question processing module is used to preprocess the input questions, including text cleaning, word segmentation, and stop word removal;

[0012] The intent recognition module is used to identify key entities in the input question and make judgments and classifications;

[0013] The answer generation module is used to call the database to answer questions based on the question category and key entities based on the GPT large language model;

[0014] The storage module is used to store interaction records, which are used for secondary training of the GPT large language model.

[0015] Furthermore, in the interactive module, privacy preference information is collected through a questionnaire, which includes the type of data that the user is willing to share, the sensitivity level of the data, and the degree of acceptance of the risk of privacy leakage. The questionnaire is a scoring system, and the user's comprehensive score is the average of all item scores. The data uploaded by the user is divided into several privacy protection levels according to the comprehensive score.

[0016] Furthermore, the data processing module is also provided with a slicing unit for slicing the document content according to the document layout, paragraph information and a preset maximum length of the fragment.

[0017] Furthermore, the data processing module is also used to classify data into two categories: tables and texts according to the document content. Each level of database includes a structured database and an unstructured database. The structured database is used to store table data, and the unstructured database is used to store text data.

[0018] Furthermore, in the disturbance module, the disturbance amount is noise, and the noise conforms to one or more of Laplace distribution, Gaussian distribution and exponential distribution.

[0019] Furthermore, the GPT large language model is fine-tuned using the thought chain technology.

[0020] Furthermore, the intent recognition module is provided with a semantic analysis unit, which is used to accurately identify key entities in the input question using semantic analysis technology, and at the same time uses the GPT large language model as a backup method for key entity recognition; if the semantic analysis unit fails to identify the key entity in the input question, the input question and the logical process of the semantic analysis unit are input into the GPT large language model together, and the output result is then passed to the semantic analysis unit for key entity recognition.

[0021] Furthermore, in the intent recognition module, question classification includes direct generation class, unstructured query class and structured query class.

[0022] Furthermore, the answer generation module is also provided with a retrieval unit for retrieving data in a database.

[0023] Furthermore, the answer generation module is also provided with a construction unit, which is used to construct corresponding prompts according to the key entities of the question and the data retrieved by the retrieval unit, and input them into the GPT large language model.

[0024] The beneficial effects of the basic scheme are: 1. The present invention collects users' privacy preference information and divides users into different privacy protection levels based on this information. On the one hand, the model can ensure that while providing knowledge question and answer services, it fully respects and protects users' privacy, enhances the privacy protection capabilities of the data, and reduces the risk of privacy leakage. On the other hand, according to the user's privacy protection level, the amount of disturbance of the database by differential privacy can be adjusted. Compared with the existing method of randomly adding disturbance, the impact of the disturbance on the database can be appropriately reduced, thereby effectively improving the accuracy of the knowledge question and answer model.

[0025] 2. The input questions are classified through the intention recognition module, and the database is divided into structured and unstructured databases. The text and table data are queried by category, which improves the precision and flexibility of the retrieval unit and thus achieves the accuracy of the knowledge question and answer model.

[0026] 3. By fine-tuning the GPT large language model using the thought chain technology, the large model is provided with some thinking example prompts about the problem's demonstration input, process, and output, so that the model can simulate the human problem-solving process to a certain extent, which helps users better understand the source and derivation process of the answer, enhance their trust in the answer, and improve the accuracy and reliability of the answer.

[0027] 4. By collecting and analyzing the model's thinking chain and answer quality in the problem-solving process through the storage module, the model's performance and reasoning ability can be further optimized, which will help the model better cope with more complex and sophisticated problems in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a system diagram of a knowledge question-answering model based on differential privacy and large-model thinking chain in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is further described in detail through specific implementation methods:

[0030] Example 1

[0031] The embodiment is basically as shown in the attached Figure 1 Shown: A knowledge question-answering model based on differential privacy and large model thinking chain, including:

[0032] The interactive module is used to collect the privacy preference information of users before uploading data through the front-end interface, and divide the data uploaded by users into several privacy protection levels; the privacy preference information is collected in the form of a questionnaire, which includes the type of data that users are willing to share, the sensitivity level of the data and the degree of acceptance of the risk of privacy leakage. The questionnaire is a scoring system, and the user's comprehensive score is the average of all item scores. The data uploaded by users is divided into several privacy protection levels according to the comprehensive score.

[0033] The data processing module is used to collect documents, extract document content, build a database, and divide the database into several levels according to the user's privacy protection level; the data processing module is also provided with a slicing unit, which is used to slice the document content according to the document layout, paragraph information and preset maximum length of the fragment; the data processing module is also used to divide the data into two categories: tables and texts according to the document content. Each level of the database includes a structured database and an unstructured database. The structured database is used to store table data, and the unstructured database is used to store text data.

[0034] The perturbation module is used to add perturbations to the database. Each level of the database corresponds to a different perturbation. The higher the database privacy protection level, the greater the perturbation, and the perturbation parameter satisfies a linear relationship with the database level. The perturbation is noise, and the noise conforms to one or more of the Laplace distribution, Gaussian distribution, and exponential distribution.

[0035] The question processing module is used to preprocess the input questions, including text cleaning, word segmentation and removal of stop words.

[0036] The intention recognition module is used to identify the key entities in the input question and make judgments and classifications, including direct generation class, unstructured query class and structured query class; the intention recognition module is equipped with a semantic analysis unit, which is used to accurately identify the key entities in the input question using semantic analysis technology, and uses the GPT large language model as a backup method for key entity recognition; if the semantic analysis unit fails to identify the key entity in the input question, the input question and the logical process of the semantic analysis unit are input into the GPT large language model together, and the output result is passed to the semantic analysis unit for key entity recognition.

[0037] The answer generation module is used to call the database for answering based on the GPT large language model according to the question category and key entities; the GPT large language model is fine-tuned using the thinking chain technology; the answer generation module also has a retrieval unit for retrieving data from the database; the answer generation module also has a construction unit for constructing corresponding prompts based on the key entities of the question and the data retrieved by the retrieval unit, and inputting them into the GPT large language model.

[0038] The storage module is used to store interaction records, which are used for secondary training of the GPT large language model.

[0039] The specific implementation process is as follows:

[0040] Taking the knowledge question answering model in the financial field as an example, the operation includes the following steps:

[0041] Collect privacy preference information: Conduct questionnaire surveys on corporate users in related financial fields by creating websites and other front-end interfaces, including the types of data users are willing to share (company basic information, company employee information, company financial statements, etc.), data sensitivity levels (1-10, the higher the level, the higher the sensitivity) and the degree of acceptance of privacy leakage risks (1-10, the higher the level, the higher the acceptance). Collect questionnaire results through interactive modules, take the average of the comprehensive scores, and divide the data uploaded by users into several privacy protection levels. Here, the privacy protection levels are divided into 5 levels, and the higher the level, the higher the privacy protection level.

[0042] Data collection and processing: Financial documents are collected through the data processing module, including formats such as txt, word, pdf and excel. Due to the large differences in the length of document content, longer documents cannot be directly input into the large model, so the slicing unit is used to slice the documents, and the data is divided into two categories: tables and texts according to the document content to form a database; according to the privacy protection level of the user providing the data, the database is divided into 5 levels, and each level of the database includes a structured database and an unstructured database. The structured database is used to store table data, and the unstructured database is used to store text data.

[0043] Differential privacy processing: Through the perturbation module, perturbation noise is added to each level of the database. The probability of the perturbation noise added in this embodiment satisfies the relationship: P(x) = e ε +δ, where x is the database level, e ε and δ are disturbance noises, ε is a noise parameter, and the similarity and difference of the algorithm output on each level of the database are ensured by controlling ε. x is linearly related to ε. At the same time, in this embodiment, the distribution of (ε, δ) in the database conforms to the Laplace distribution.

[0044] Fine-tuning the large language model: Use the chain of thought technology COT to fine-tune the GPT large language model. The Prompt prompt of the chain of thought technology CoT is a triple consisting of input, chain of thought and output (input, chain of thought, output); the input is recorded as the question instance Q, the chain of thought is a series of intermediate natural language reasoning steps to obtain the final output result, recorded as R, and the output is the expected answer A, then the prompt is P = {(Q1, A1, R1), ..., (Q k ,A k ,R k )}, it can be seen that different expected answers can be obtained at different stages of reasoning. Collect common questions Q, reasoning steps R and expected answers A in the financial field, build prompts to train the large language model, and complete fine-tuning.

[0045] Input question processing: The input question is preprocessed through the question processing module, and the key entity recognition is performed through the semantic analysis unit in the intent recognition module. The question recognition is classified into direct generation class, unstructured query class and structured query class; if the semantic analysis unit fails to recognize the key entity in the input question, the input question and the logical process of the semantic analysis unit are input into the GPT large language model together, and the output result is passed to the semantic analysis unit for key entity recognition.

[0046] Answer generation and feedback: Through the answer generation module, answers are given according to the categories:

[0047] Direct generation: Based on the key entities, the corresponding prompts are directly constructed through the construction unit, and input into the GPT big model, which then generates the question response;

[0048] Unstructured query: The retrieval unit retrieves relevant text fragments from the unstructured database based on key entities, and then the construction module constructs corresponding prompts based on key entities and relevant text fragments, which are input into the large model to generate question responses;

[0049] Structured query class: The construction unit constructs instructions for generating SQL query statements based on key entities, and then the big model generates corresponding SQL statements. The retrieval unit retrieves the statements in the structured database to obtain corresponding data. Then, the construction unit constructs corresponding prompts based on key entities and retrieved data, and inputs them into the big model to generate question responses;

[0050] During the question answering process, the interaction records are stored through the storage module and used in the secondary training of the GPT large language model to further optimize the performance and reasoning ability of the model.

[0051] Example 2

[0052] Different from the above embodiment, a test data set of financial knowledge question and answer is constructed to verify the performance of the question and answer model. The data set type and quantity are as follows:

[0053] Table 1 Test set types and numbers

[0054] Question and answer type Specific categories Number of questions Unstructured Q&A Direct Generation 30 Unstructured Q&A Unstructured query 50 Structured Q&A Structured Query 60

[0055] Evaluation criteria: The accuracy is used to evaluate the performance of the question-answering model. The relationship is expressed as: ACC% = N C / N T ×100, where N C is the number of questions answered correctly, N T For all question numbers.

[0056] The test results are shown in the following table:

[0057] Table 2 Test set accuracy

[0058]

[0059] The results show that the accuracy of the direct generation class is 96.5%, the accuracy of the unstructured query class is 90.3%, the accuracy of the structured query is 85.2%, and the overall accuracy is 90.7%, which can effectively solve the basic question-and-answer needs related to the financial field. Compared with the existing knowledge question-and-answer model based on differential privacy and large model thinking chain, this method can effectively improve the accuracy of question-and-answer while ensuring user privacy.

[0060] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0061] Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0062] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A knowledge question-answering model based on differential privacy and large-model thinking chain, characterized by: include: The interactive module is used to collect the privacy preference information of users before uploading data through the front-end interface, and classify the data uploaded by users into several privacy protection levels; The data processing module is used to collect documents, extract document contents, build a database, and divide the database into several levels according to the privacy protection level; The perturbation module is used to add perturbations to the database. Each level of database corresponds to a different perturbation amount. The higher the database privacy protection level, the greater the perturbation amount, and the perturbation amount parameter satisfies a linear relationship with the database level. The question processing module is used to preprocess the input questions, including text cleaning, word segmentation, and stop word removal; The intent recognition module is used to identify key entities in the input question and make judgments and classifications; The answer generation module is used to call the database to answer questions based on the question category and key entities based on the GPT large language model; The storage module is used to store interaction records, which are used for secondary training of the GPT large language model.

2. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 1 is characterized by: In the interactive module, privacy preference information is collected through a questionnaire, which includes the type of data that the user is willing to share, the sensitivity level of the data, and the degree of acceptance of the risk of privacy leakage. The questionnaire is a scoring system, and the comprehensive score is the average of all item scores. The data uploaded by the user is divided into several privacy protection levels according to the comprehensive score.

3. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 2 is characterized by: The data processing module is also provided with a slicing unit for slicing the document content according to the document layout, paragraph information and a preset maximum length of a segment.

4. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 3 is characterized by: The data processing module is also used to classify data into two categories: tables and texts according to the document content. Each level of database includes a structured database and an unstructured database. The structured database is used to store table data, and the unstructured database is used to store text data.

5. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 4 is characterized by: In the disturbance module, the disturbance amount is noise, and the noise conforms to one or more of Laplace distribution, Gaussian distribution and exponential distribution.

6. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 5 is characterized by: The GPT large language model is fine-tuned using the thought chain technology.

7. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 6 is characterized by: The intent recognition module is equipped with a semantic analysis unit, which is used to accurately identify key entities in input questions using semantic analysis technology, and uses the GPT large language model as a backup method for key entity recognition; If the semantic analysis unit fails to identify the key entities in the input question, the input question and the logical process of the semantic analysis unit are input into the GPT large language model together, and the output result is passed to the semantic analysis unit for key entity recognition.

8. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 7 is characterized by: In the intent recognition module, question classification includes direct generation class, unstructured query class and structured query class.

9. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 8, characterized in that: The answer generation module is also provided with a retrieval unit for retrieving data in a database.

10. The knowledge question-answering model based on differential privacy and large model thinking chain according to claim 9, characterized in that: The answer generation module also has a construction unit, which is used to construct corresponding prompts based on the key entities of the question and the data retrieved by the retrieval unit, and input them into the GPT large language model.

Images