Large language model natural language processing method based on causal knowledge retrieval

Abstract

The invention discloses a big language model natural language processing method and device based on causal knowledge retrieval, a knowledge migration method and device, a storage medium, equipment and a computer program product. Obtaining question and answer texts respectively corresponding to each answer text under the question text; according to the corresponding question and answer text and the corresponding answer text, determining a causal evaluation value of the question text for each answer text; and determining the answer text corresponding to the maximum causal evaluation value in the multiple causal evaluation values as a target answer text matched with the question text. Based on the method, the actual contribution of the answer text is quantified through the causal evaluation value, the problem that the reasoning result is inaccurate in the natural language processing process is solved, meanwhile, the causal reasoning ability is migrated to the lightweight model through the knowledge migration method, and the reasoning accuracy is improved. Therefore, the method still has the efficient operation capability in a low-resource environment.

Description

Technical Field

[0001] This application belongs to the field of natural language processing, and particularly relates to a natural language processing method, a knowledge transfer method, a device, a storage medium, a device, and a computer program product for large language models based on causal knowledge retrieval. Background Art

[0002] In the wide application of Natural Language Processing (NLP), the ability of machines to understand and generate natural language text becomes crucial. These applications include scenarios such as dialogue systems, text summarization, open-domain question answering, and information retrieval. How to retrieve accurate answers from a vast external knowledge base and effectively reduce the misleading nature of the generated content is a major challenge.

[0003] In the prior art, the Retrieval-augmented Generation (RAG) method enhances the knowledge ability of language models by combining Information Retrieval (IR) and Natural Language Generation (NLG). These methods usually rely on semantic similarity search to determine relevant knowledge.

[0004] However, semantic similarity retrieval easily leads to retrieving information that is superficially similar but actually irrelevant to the query, and the finally generated answers are inaccurate or even misleading. In addition, existing multi-step RAG (such as Chain-of-Thought) methods need to repeatedly call the Large Language Model (LLM) for query splitting and re-retrieval, resulting in high computational complexity and long query latency, which affects the overall efficiency of the system. Summary of the Invention

[0005] This application aims to provide a natural language processing method, a knowledge transfer method, a device, a storage medium, a device, and a computer program product for large language models based on causal knowledge retrieval, which at least solves the problems of low inference efficiency and inaccurate results in the process of natural language processing.

[0006] In a first aspect, an embodiment of this application discloses a natural language processing method for large language models based on causal knowledge retrieval, including: Splicing the question text with multiple answer texts to be matched respectively to obtain question-answer texts corresponding to each answer text under the question text; Determine the causal evaluation value of each of the response texts for the corresponding question text; the causal evaluation value is positively correlated with the conditional probability that the content of the response text matches the question text in the case where the response text is the feedback result of the question text. Determine the response text corresponding to the largest causal evaluation value among the multiple causal evaluation values as the target response text that matches the question text.

[0007] In a second aspect, an embodiment of the present application also discloses a knowledge transfer method for a large language model based on causal knowledge retrieval, including: Obtain multiple question training texts; Input the multiple question training texts into a large language model based on causal knowledge retrieval respectively to determine, from multiple response training texts, the first target response training text corresponding to each question training text, and the target causal evaluation value of each question training text for each first target response training text; the target causal evaluation value is used to represent the conditional probability that the content of the first target response training text matches the question training text in the case where the first target response training text is the feedback result of the question training text. Train a lightweight large language model according to multiple groups of corresponding question training texts, first target response training texts, and target causal evaluation values to obtain the trained lightweight large language model.

[0008] In a third aspect, an embodiment of the present application also discloses a natural language processing device for a large language model based on causal knowledge retrieval, including: A splicing module for splicing the question text with multiple response texts to be matched respectively to obtain a question-answer text corresponding to each response text under the question text. An evaluation module for determining the causal evaluation value of each response text for the question text according to the corresponding question-answer text and response text; the causal evaluation value is positively correlated with the conditional probability that the content of the response text matches the question text in the case where the response text is the feedback result of the question text. A selection module for determining the response text corresponding to the largest causal evaluation value among the multiple causal evaluation values as the target response text that matches the question text.

[0009] In a fourth aspect, an embodiment of the present application also discloses a knowledge transfer device for a large language model based on causal knowledge retrieval, including: A training set module for obtaining multiple question training texts; A migration set module, configured to input multiple pieces of the question training text into a large language model based on causal knowledge retrieval respectively, so as to determine first target answer training texts respectively corresponding to each piece of the question training text from multiple pieces of answer training text, and target causal evaluation values of the question training text for each of the first target answer training texts; the target causal evaluation value is used to represent the conditional probability that the content of the first target answer training text matches the question training text in the case that the first target answer training text is a feedback result of the question training text. A transfer training module, configured to train a lightweight large language model according to multiple groups of corresponding question training text, first target answer training text and target causal evaluation values, so as to obtain the trained lightweight large language model.

[0010] In a fifth aspect, an embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps described in the first aspect or the second aspect are implemented.

[0011] In a sixth aspect, an embodiment of the present application further discloses an electronic device, including a processor, a memory, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, the steps described in the first aspect or the second aspect are implemented.

[0012] In a seventh aspect, an embodiment of the present application further discloses a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the steps described in the first aspect or the second aspect are implemented.

[0013] In summary, in the embodiment of the present application, by determining the causal evaluation value of the question text for each answer text, not only relying on semantic similarity, but also being able to quantify the actual help degree of the answer text to the question text, thus avoiding misleading information; and then according to the causal evaluation value, selecting the target answer text corresponding to the maximum causal evaluation value to ensure that the most relevant text is given priority and then selected, thereby significantly improving the accuracy of generating answers. Therefore, based on the method of the embodiment of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it transcends the traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of generating answers. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiment of the present application, the causal reasoning ability is transferred to a lightweight model through the knowledge transfer method of the present application, solving the problem of high computational cost in the prior art, and enabling the method to still have the ability to operate efficiently in a low-resource environment. Description of the Drawings

[0014] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 is a flowchart of the steps of a natural language processing method for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 2 is a flowchart of the steps of another natural language processing method for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 3 is a flowchart of the steps of a knowledge transfer method for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 4 is a flowchart of the steps of another knowledge transfer method for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 5 is a schematic structural diagram of a natural language processing device for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 6 is a schematic structural diagram of a knowledge transfer device for a large language model based on causal knowledge retrieval provided by an embodiment of the present application; Figure 7 is a block diagram of an electronic device provided by an embodiment of the present application; Figure 8 is a block diagram of another electronic device provided by an embodiment of the present application. Detailed Embodiments

[0015] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully conveyed to those skilled in the art.

[0016] Figure 1 is a natural language processing method for a large language model based on causal knowledge retrieval provided by an embodiment of the present application, specifically including the following steps: Step 101, splice the question text with multiple answer texts to be matched respectively to obtain question-answer texts corresponding to each answer text under the question text.

[0017] In some embodiments of the present application, in order to directly compare the question text with potential answer texts and form a question-and-answer text rich in context, the question text will be concatenated with multiple answer texts to be matched to obtain question-and-answer texts corresponding to each answer text under the question text. A question-and-answer text refers to a combined text generated based on the question text and the answer text. In this way, a set of more contextually related text pairs can be generated, improving the accuracy of subsequent causal evaluation.

[0018] In a specific example, a user inputs a question text, such as "What is an embedded model?" into the system. The system concatenates the question text with multiple answer texts, such as "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources. They are optimized to perform efficient inference with limited computing power".

[0019] Step 102: Determine the causal evaluation value of the question text for each answer text according to the corresponding question-and-answer text and the answer text.

[0020] Among them, the causal evaluation value is positively correlated with the conditional probability that the content of the answer text matches the question text in the case where the answer text is the feedback result of the question text.

[0021] In some embodiments of the present application, in order to further improve the accuracy of answer generation by quantifying the actual helpfulness of the answer text to the question text, the causal evaluation value of the question text for each answer text will be determined according to the corresponding question-and-answer text and the answer text. The causal evaluation value is used to measure whether the answer text is actually helpful in the feedback result of the question text. In this way, the conditional probability that each answer text matches the content of the question text can be obtained, thereby determining its causal evaluation value.

[0022] In a specific example, a user inputs the question text "What is an embedded model?" into the system. The system analyzes the previously generated question-and-answer text pairs and calculates the causal evaluation value of each answer text. For example, the system evaluates the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" and obtains its causal evaluation value of 0.85. This indicates that the answer text has a high causal correlation with the question text when explaining the embedded model, thereby improving the accuracy of generating answers.

[0023] Step 103: Determine the answer text corresponding to the maximum causal evaluation value among the multiple causal evaluation values as the target answer text that matches the question text.

[0024] In some embodiments of the present application, in order to ensure that the most causally relevant answer text is preferentially selected to improve the accuracy of the answer, the answer text corresponding to the largest causal evaluation value among multiple causal evaluation values is determined as the target answer text that matches the question text. In this way, the answer text that best matches the question text can be screened out, ensuring that the generated answer has higher reliability and accuracy.

[0025] In a specific example, the user inputs the question text "What is an embedded model?" into the system. The system has calculated the causal evaluation values of multiple answer texts. For example, the causal evaluation value of answer text "A" is 0.85, and the causal evaluation value of answer text "B" is 0.65. The system determines the answer text "A" with the highest causal evaluation value as the target answer text that matches the question text.

[0026] In summary, in the embodiments of the present application, by determining the causal evaluation value of the question text for each answer text, not only relying on semantic similarity, but also being able to quantify the actual helpfulness of the answer text to the question text, thus avoiding misleading information; and then according to the causal evaluation value, selecting the target answer text corresponding to the largest causal evaluation value to ensure that the most relevant text is preferentially considered and then selected, thereby significantly improving the accuracy of the generated answer. Therefore, based on the method of the embodiments of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it surpasses the traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of the generated answer. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiments of the present application, through the knowledge transfer method of the present application, the causal reasoning ability is transferred to the lightweight model, solving the high computational cost problem in the prior art, and enabling the method to still have the ability to operate efficiently in a low-resource environment.

[0027] Figure 2 It is another natural language processing method for large language models based on causal knowledge retrieval provided by the embodiments of the present application, specifically including the following steps: Step 201, according to the preset similarity conditions, perform similarity matching between the question text and multiple texts to obtain multiple answer texts to be matched.

[0028] In some embodiments of the present application, in order to preliminarily screen out answer texts that are semantically similar to the question text for further evaluating their causal relevance in subsequent steps, according to the preset similarity conditions, similarity matching is performed between the question text and multiple texts to obtain multiple answer texts to be matched, thereby reducing the computational burden of subsequent matching. The similarity condition refers to the standard for measuring the semantic similarity degree between two texts. In this way, a set of preliminary candidate answer texts that are semantically related to the question text can be generated, providing a basis for subsequent causal evaluation.

[0029] In a specific example, a user inputs the query text "What is an embedded model?" into the system. The system matches the query text with multiple texts in the knowledge base according to preset similarity conditions, such as Word Embedding or vector space model, and filters out a set of response texts that are semantically similar to the query text. For example, the response texts matched by the system include "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources". Through this similarity matching, the system can filter out relevant response texts for the subsequent causal evaluation step, improving the accuracy of the finally generated answer.

[0030] Step 202: Concatenate the query text with each of the multiple response texts to be matched respectively, so as to obtain the Q&A texts corresponding to each response text under the query text.

[0031] The method shown in this step has been described in step 101, and will not be elaborated here.

[0032] Step 203: Determine the causal evaluation value of the query text for each response text according to the corresponding Q&A text and response text.

[0033] Among them, the causal evaluation value is positively correlated with the conditional probability of the content matching between the response text and the query text in the case where the response text is the feedback result of the query text.

[0034] The method shown in this step has been described in step 102, and will not be elaborated here.

[0035] Specifically, in order to improve the retrieval accuracy and reduce the self-confirmation bias existing in traditional methods, this application proposes a retrieval enhancement method based on Causal Inference Score (CIS). CIS aims to quantify the causal relationship between the query text and the response text by evaluating the degree of entailment of the document content to the query through an autoregressive language model.

[0036] In some specific embodiments, the calculation formula of CIS is as follows: , Among them, is the parameter of the language model, represents the entailment probability of the LLM for the response text under given the query document , while represents the LLM for The inherent familiarity, and all these data can be obtained when... Through this calculation method, the causal impact of the text on the query can be quantified, improving the accuracy and reliability of the retrieval results.

[0037] At this time, when the CIS score is positive, it indicates a strong correlation between the query and the document, suggesting a possible causal relationship; a CIS score of zero indicates that the query and the document are independent of each other; a negative CIS score means that the probability of the query and the document appearing simultaneously is low. Therefore, the CIS score can be used to replace the similarity score in the traditional retrieval model, and the top k documents with the highest CIS scores are used as the knowledge support for generating answers by the large language model (LLM).

[0038] To measure the confidence of the LLM in containing it, the conditional probability can be calculated using a pre-trained language model (PLM). Specifically, the document is represented as a word sequence and concatenated with , and then the PLM is used with as the prefix to gradually predict the generation probability of each token in .

[0039] The conditional probability calculation formula is as follows:

[0040] Meanwhile, and can be organized into a prompt in the form of "question - answer", such as "Q: A: " to more clearly measure the matching degree of the document content to the query.

[0041] Since the language model is more familiar with some text fragments that frequently appear in the knowledge base, its confidence in containing these fragments may be overestimated, resulting in self - confirmation bias. This bias affects the accuracy of the retrieval, so the confidence in containment is further normalized by the generation probability of the text itself to correct this bias. Specifically, the generation probability of the text is calculated from the conditional probabilities of its internal word sequence, that is, calculating the generation probability of each word under the condition of its previous word sequence , and its calculation formula is as follows: .

[0042] Based on the above process, optionally, step 203 includes the following sub - steps: Sub-step 2031: Determine the conditional probability data of the Q&A text matching the content of the question text and the answer probability data of the answer text being the feedback result of the question text in the case where the answer text is the feedback result of the question text.

[0043] In some embodiments of the present application, in order to accurately quantify the content matching between the answer text and the question text, the conditional probability data of the Q&A text matching the content of the question text and the answer probability data of the answer text being the feedback result of the question text will be determined in the case where the answer text is the feedback result of the question text. The conditional probability data of Q&A refers to the conditional probability that the Q&A text matches the content of the question text given the question text, and the answer probability data refers to the probability that the answer text is the feedback result of the question text given the question text. In this way, the system will be able to quantify the content matching of the answer text and provide basic data support for subsequent causal evaluation.

[0044] In a specific example, after inputting the question text "What is an embedded model?" into the system. The system analyzes multiple answer texts and determines the conditional probability data of Q&A and the answer probability data of each answer text in the case where the question text is its feedback result. For example, the system analyzes the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" and calculates its conditional probability data of Q&A as 0.80 and the answer probability data as 0.90. In this way, the system can quantify the content matching between the answer text and the question text and provide accurate data support for subsequent causal evaluation.

[0045] Optionally, sub-step 2031 includes the following sub-steps: Sub-step 20311: Determine the answer token sequence of the answer text and the Q&A token sequence of the Q&A text according to the word segmentation extraction of the answer text.

[0046] Among them, the answer sequence items of the answer token sequence are composed of each sequence token in the answer text recorded in order; the Q&A sequence items of the Q&A token sequence are composed of each sequence token in the answer text recorded in order and the question-answer sequence items composed of the question text; the question-answer sequence items are the first question-answer sequence items in the Q&A sequence.

[0047] In some embodiments of the present application, in order to ensure the structured processing of the answer text and provide basic data for subsequent probability calculation, the answer token sequence of the answer text and the Q&A token sequence of the Q&A text are determined according to the token extraction of the answer text. The answer sequence items of the answer token sequence are composed of each sequence token in the answer text recorded in order; the Q&A sequence items of the Q&A token sequence are composed of each sequence token in the answer text recorded in order and the question Q&A sequence item composed of the question text, and the question Q&A sequence item is the first Q&A sequence item in the Q&A sequence. Tokenization is the process of dividing text into independent words or subsequences. After performing this step, structured sequence data for subsequent probability calculation can be generated.

[0048] In a specific example, the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" is input into the system. The system tokenizes the answer text to obtain the answer token sequence: "embedded", "model", "is", "a", "in", "computing resources", "limited", "environment", "runs", "lightweight", "neural network", "model". At the same time, the system concatenates the question text "What is an embedded model?" with the tokenization result of the answer text to generate the Q&A token sequence: "What is an embedded model", "embedded", "model", "is", "a", "in", "computing resources", "limited", "environment", "runs", "lightweight", "neural network", "model". In this way, the system can generate a structured token sequence to provide basic data for subsequent conditional probability calculation.

[0049] Sub-step 20312: respectively determine the answer previous item sequence of the answer text and the Q&A previous item sequence of the Q&A text according to the extended selection starting from the first item of the answer token sequence and the Q&A token sequence.

[0050] Among them, the sequence items of the answer previous item sequence and the Q&A previous item sequence correspond to the sequence items of the Q&A token sequence in order one by one.

[0051] In some embodiments of the present application, in order to ensure the detailed conditional probability calculation of the answer text and the Q&A text by gradually expanding the sequence, the answer previous item sequence of the answer text and the Q&A previous item sequence of the Q&A text are determined respectively according to the extended selection starting from the first item of the answer token sequence and the Q&A token sequence. The sequence items of the answer previous item sequence and the Q&A previous item sequence correspond to the sequence items of the Q&A token sequence in order one by one. The answer previous item sequence and the Q&A previous item sequence refer to the partial sequences that gradually increase starting from the first item on the basis of the token sequence. In this way, the system can generate extended sequence data for conditional probability calculation to provide an accurate basis for subsequent evaluation.

[0052] In a specific example, after inputting the response text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" into the system, the system tokenizes the response text and generates a response token sequence and a Q&A token sequence. Then, based on the response token sequence and the Q&A token sequence, the system gradually expands from the first item to generate a response prefix sequence and a Q&A prefix sequence. For example, the response prefix sequence can be "embedded", "embedded model", "embedded model is", etc., and the Q&A prefix sequence can be "What is an embedded model embedded", "What is an embedded model embedded model", "What is an embedded model embedded model is", etc. In this way, the system can generate detailed expanded sequence data, providing an accurate basis for subsequent conditional probability calculations.

[0053] Sub-step 20313: Record the conditional probability of each sequence item of the Q&A token sequence under the sequence item of the Q&A prefix sequence corresponding to the sequence item of the Q&A token sequence as a component value of the Q&A conditional probability data, and record the conditional probability of each sequence item of the response token sequence under the sequence item of the response prefix sequence corresponding to the sequence item of the response token sequence as a component value of the response probability data.

[0054] In some embodiments of the present application, in order to calculate the probability of each sequence item under the condition of its prefix sequence, so as to quantify the content matching of the Q&A text and the response text, the conditional probability of each sequence item of the Q&A token sequence under the sequence item of the Q&A prefix sequence corresponding to the sequence item of the Q&A token sequence is recorded as a component value of the Q&A conditional probability data, and the conditional probability of each sequence item of the response token sequence under the sequence item of the response prefix sequence corresponding to the sequence item of the response token sequence is recorded as a component value of the response probability data. Conditional probability refers to the probability of the current sequence item occurring given the prefix sequence. After performing this step, the system can quantify the conditional probability of each sequence item, providing detailed probability data for subsequent causal evaluation.

[0055] In a specific example, the query text "What is an embedded model?" and the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" are input into the system. The system tokenizes and expands these texts to generate an answer token sequence, a Q&A token sequence, and corresponding previous item sequences. Then, the system calculates the conditional probability of each sequence item in the Q&A token sequence under the condition of the corresponding Q&A previous item sequence and records it as a component value of the Q&A conditional probability data. For example, the conditional probability of the Q&A token sequence "embedded model" is 0.75 and is recorded as a component value of the Q&A conditional probability data. The system also calculates the conditional probability of each sequence item in the answer token sequence under the condition of the corresponding answer previous item sequence and records it as a component value of the answer probability data. In this way, the system can generate detailed conditional probability data, providing an accurate basis for subsequent causal evaluation.

[0056] Sub-step 2032: Determine the causal evaluation value of the answer text based on the logarithm of the ratio of the Q&A probability value and the answer probability value determined according to the Q&A conditional probability data and the answer probability data.

[0057] Among them, the Q&A probability value is the probability value of the conditional probability that the Q&A text matches the content of the query text when the answer text is the feedback result of the query text; the answer probability value is the probability value of the probability that the answer text is the feedback result of the query text.

[0058] In some embodiments of the present application, in order to quantify the actual contribution degree of the answer text by calculating the causal correlation between the answer text and the query text, the causal evaluation value of the answer text is determined based on the logarithm of the ratio of the Q&A probability value and the answer probability value determined according to the Q&A conditional probability data and the answer probability data. The Q&A probability value is the probability value of the conditional probability that the Q&A text matches the content of the query text when the answer text is the feedback result of the query text; the answer probability value is the probability value of the probability that the answer text is the feedback result of the query text. In this way, the system will obtain the causal evaluation value of the answer text, providing a basis for subsequent selection of the optimal answer text.

[0059] In a specific example, after inputting the question text "What is an embedded model?" into the system, the system analyzes multiple answer texts and determines the Q&A probability value and the answer probability value of each answer text in the case where the question text is its feedback result. For example, for the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources", the system calculates its Q&A probability value to be 0.80 and the answer probability value to be 0.90. The system further calculates the ratio of these two values and takes the logarithm to determine the causal evaluation value of this answer text to be 0.08. In this way, the system can quantify the causal correlation between each answer text and the question text, so as to select the best answer text and improve the accuracy of generating answers.

[0060] Optionally, sub-step 2032 includes the following sub-steps: Sub-step 20321, perform a logarithm operation on each component value of the Q&A conditional probability data and the answer probability data to obtain the Q&A scoring data of the Q&A text and the answer scoring data of the answer text.

[0061] In some embodiments of the present application, in order to convert the conditional probability data into scoring data, so as to better quantify the correlation between the Q&A text and the answer text, a logarithm operation is performed on each component value of the Q&A conditional probability data and the answer probability data to obtain the Q&A scoring data of the Q&A text and the answer scoring data of the answer text. The logarithm operation refers to taking the logarithm of each component value to more intuitively compare and analyze the probability data. The advantage of the logarithm operation is that the complex floating-point multiplication and division operations can be converted into addition and subtraction operations. In this way, the system can obtain more easily processed and analyzed scoring data, providing a quantitative basis for subsequent causal evaluation.

[0062] In a specific example, after inputting the question text "What is an embedded model?" and the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" into the system. The system calculates the Q&A conditional probability data and the answer probability data of these texts. For example, a component value of a certain Q&A conditional probability data is 0.80, and the corresponding component value of the answer probability data is 0.90. The system performs a logarithm operation on these two component values respectively to obtain the Q&A scoring data and the answer scoring data. For example, the logarithm operation result is Q&A scoring data -0.22 and answer scoring data -0.15. In this way, the system can convert the conditional probability data into scoring data, providing a quantitative basis for subsequent causal evaluation.

[0063] Sub-step 20322, determine the difference between the sum of each component value of the Q&A scoring data and the sum of each component value of the answer scoring data as the causal evaluation value.

[0064] In some embodiments of the present application, in order to quantify the causal relevance of the answer text to the question text by comparing the Q&A scoring data and the answer scoring data, the difference between the sum of each component value of the Q&A scoring data and the sum of each component value of the answer scoring data is determined as the causal evaluation value of the answer text. The Q&A scoring data and the answer scoring data are respectively obtained by taking the logarithm of the Q&A conditional probability data and the answer probability data. In this way, the causal evaluation value of each answer text can be obtained, providing a basis for selecting the most relevant answer text.

[0065] In a specific example, after inputting the question text "What is an embedded model?" and the answer text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" into the system. The Q&A conditional probability data and the answer probability data will be calculated first, and then the logarithm operation will be performed on them to obtain the Q&A scoring data and the answer scoring data. For example, the sum of the component values of the Q&A scoring data is 1.5, and the sum of the component values of the answer scoring data is -1.8. The system compares the sums of these two data, obtains the difference of 3.3, and determines it as the causal evaluation value. In this way, the system can quantify the causal relevance of each answer text to the question text and improve the accuracy of selecting the best answer text.

[0066] Optionally, as an alternative to sub-steps 2031 and 2032 of the present application, the present application can also obtain the causal evaluation value in the following way: Optionally, in some alternative solutions of the present application, the causal evaluation value can also be obtained based on the method of Pointwise Mutual Information (PMI): Pointwise Mutual Information (PMI) is a statistical method for measuring the co-occurrence correlation between two random variables (in the present invention, and ). The calculation formula of PMI is as follows: , This formula measures the probability of co-occurrence of the query and the document relative to the probability of their independent occurrence . The larger the PMI value, the stronger the co-occurrence relationship between and , and vice versa, indicating a higher degree of independence between the two. In information retrieval tasks, PMI can be used to measure the relevance of a document to a query without relying on the LLM to calculate the conditional probability , so the computational overhead can be significantly reduced. In addition, PMI relies on statistical data and can be pre-computed using a large-scale text corpus, making the retrieval process more efficient.

[0067] Optionally, in some alternative solutions of the present application, the causal evaluation value can also be determined based on the method of conditional entropy: Conditional Entropy is a method of measuring the information provided. Its calculation formula is as follows: , In the framework of information theory, conditional entropy reflects the uncertainty of given the . In other words, the lower the conditional entropy, the stronger the explanatory ability of the , and thus it is more suitable as high-quality retrieval results. In the information retrieval task, the conditional entropy method can be used as an alternative scoring method to measure the causal contribution of a document to a query. Different from the PMI method, conditional entropy pays more attention to the reduction of information uncertainty rather than just the co-occurrence frequency. Therefore, it is particularly suitable for retrieval tasks that require causal explanations, such as causal reasoning, legal text analysis, medical literature recommendation, etc.

[0068] Optionally, in some alternative solutions of the present application, the causal evaluation value can also be obtained based on Bayesian Inference: Bayesian Inference provides a method for calculating causal influence based on prior probability and posterior probability. In the retrieval task, the Bayesian formula can be used to calculate the and causal influence relationship between: , where: represents the posterior probability of the occurrence of knowledge given , that is, the causal relevance between the document and the query ; represents the conditional probability of the occurrence of the query given the knowledge , that is, the ability of the document to explain the query ; is the prior probability of the document , indicating the popularity of this document in the overall corpus; is The prior probability of occurrence can usually be estimated from user logs or historical query data of search engines. The advantage of this method is that it can combine prior information with conditional probability, thus being applicable to different types of retrieval tasks. For example, in a medical retrieval system, the prior probability of some documents is low (i.e., the documents are less cited or discussed), but due to their conditional probability is high (i.e., it has a high interpretability for a specific query), at this time, the Bayesian inference method can effectively mine high-value but low-exposure knowledge and improve the professionalism and diversity of retrieval results.

[0069] Step 204: Determine the answer text corresponding to the largest causal evaluation value among multiple causal evaluation values as the target answer text that matches the question text.

[0070] The method shown in this step has been described in step 103 and will not be elaborated here.

[0071] In summary, in the embodiments of the present application, by determining the causal evaluation value of each answer text for the question text, it not only depends on semantic similarity, but also can quantify the actual help degree of the answer text to the question text, thus avoiding misleading information; and then, according to the causal evaluation value, select the target answer text corresponding to the largest causal evaluation value to ensure that the most relevant text is considered first and then selected, thereby significantly improving the accuracy of generating answers. Therefore, based on the method of the embodiments of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it transcends the traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of generating answers. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiments of the present application, the causal reasoning ability is migrated to a lightweight model through the knowledge migration method of the present application, solving the problem of high computational cost in the prior art, and enabling the method to still have the ability to operate efficiently in a low-resource environment.

[0072] Figure 3 It is a knowledge migration method of a large language model based on causal knowledge retrieval provided by the embodiments of the present application, specifically including the following steps: Step 301: Obtain multiple question training texts.

[0073] In some embodiments of the present application, in order to provide input data for the subsequent training process, and these input data are used to train the model to improve its question-answering ability, multiple question training texts will be obtained. These texts can be carefully selected or generated and can represent various questions that users may ask. In this way, a rich and diverse set of question training texts will be available to provide the necessary data support for the subsequent training steps.

[0074] In a specific example, multiple question training texts can be input into the system. These texts may include "What is an embedded model?", "How to optimize a neural network?", "What are the applications of large language models?", etc. In this way, the system collects multiple representative question training texts, which will be used for subsequent model training to improve the Q&A ability and accuracy of the model.

[0075] Step 302: Input the multiple question training texts into the large language model based on causal knowledge retrieval respectively, so as to determine the first target answer training text corresponding to each question training text from multiple answer training texts, and the target causal evaluation value of the question training text for each first target answer training text.

[0076] The target causal evaluation value is used to represent the conditional probability that the content of the first target answer training text matches the question training text when the first target answer training text is the feedback result of the question training text.

[0077] In some embodiments of the present application, in order to screen out the most causally relevant answer text from multiple answer training texts and improve the answer accuracy of the model, the multiple question training texts will be input into the large language model based on causal knowledge retrieval respectively to determine the first target answer training text corresponding to each question training text, and calculate the target causal evaluation value of each target answer training text. The target causal evaluation value is used to represent the conditional probability of content matching between the answer training text and the question training text, reflecting their causal relevance. In this way, the causal contribution of each answer text can be quantified, providing high-quality data support for the subsequent knowledge transfer step.

[0078] In a specific example, the experimenter inputs multiple question training texts into the system, such as "What is an embedded model?" and "How to optimize a neural network?". The system inputs these question training texts into the large language model based on causal knowledge retrieval respectively, and screens out the first target answer text for each question text from the preset answer training texts. For example, for the question text "What is an embedded model?", the answer text screened out by the system is "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources". The system calculates the target causal evaluation value of this answer text to be 0.85, indicating that it has a high causal relevance when answering this question text. In this way, the system can obtain the answer text highly relevant to each question text and its causal evaluation value, providing high-quality data support for the subsequent lightweight large language model training.

[0079] Step 303: Train the lightweight large language model according to multiple groups of corresponding question training texts, first target answer training texts, and target causal evaluation values to obtain a trained lightweight large language model.

[0080] In some embodiments of the present application, in order to improve the question-answering ability and computational efficiency of the lightweight large language model by using high-quality training data, the lightweight large language model is trained according to multiple groups of corresponding question training texts, first target answer training texts, and target causal evaluation values. The training process can utilize processes such as Model Quantization or Low-Rank Adaptation (LoRA) to optimize the lightweight model with these data, enabling it to efficiently and accurately process natural language tasks. Eventually, an optimized lightweight large language model can be obtained, which can still maintain high performance in a low-computing resource environment.

[0081] In a specific example, multiple question training texts, first target answer training texts, and their corresponding target causal evaluation values can be used to train the lightweight large language model. For example, for the question training text "What is an embedded model?", the corresponding target answer text is "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources", and its target causal evaluation value is 0.85. During the training process, the system adjusts the parameters of the lightweight model according to these data, enabling it to better understand and generate accurate answers related to the question text. Finally, the system obtains a trained lightweight large language model that can still operate efficiently in a low-computing resource environment and provide high-quality natural language processing results.

[0082] In summary, in the embodiments of the present application, by determining the causal evaluation value of each answer text for the question text, not only relying on semantic similarity, but also being able to quantify the actual helpfulness of the answer text to the question text, thus avoiding misleading information; and then according to the causal evaluation value, selecting the target answer text corresponding to the maximum causal evaluation value to ensure that the most relevant text is given priority and then selected, thereby significantly improving the accuracy of generating answers. Therefore, based on the method of the embodiments of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it surpasses the traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of generating answers. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiments of the present application, the causal reasoning ability is transferred to the lightweight model through the knowledge transfer method of the present application, solving the high computational cost problem in the prior art, making the method still have the ability to operate efficiently in a low-resource environment.

[0083] Figure 4Another knowledge transfer method for large language models based on causal knowledge retrieval provided by the embodiments of the present application specifically includes the following steps: Step 401, obtain multiple question training texts.

[0084] The method shown in this step has been described in step 301 and will not be elaborated here.

[0085] Step 402, input the multiple question training texts into the large language model based on causal knowledge retrieval respectively to determine the first target answer training texts corresponding to each question training text respectively from multiple answer training texts, and the target causal evaluation values of the question training texts for each first target answer training text respectively.

[0086] Among them, the target causal evaluation value is used to represent the conditional probability of the content matching between the first target answer training text and the question training text in the case where the first target answer training text is the feedback result of the question training text.

[0087] The method shown in this step has been described in step 302 and will not be elaborated here.

[0088] Optionally, step 402 includes the following sub-steps: Sub-step 4020, store the determined first target answer training texts corresponding to each question training text respectively, and the target causal evaluation values of the question training texts for each first target answer training text respectively according to a preset data structure.

[0089] In some embodiments of the present application, in order to manage and store the target answer training texts corresponding to the question training texts and their target causal evaluation values orderly, which is convenient for subsequent model training and data analysis. The determined first target answer training texts corresponding to each question training text respectively, and the target causal evaluation values of the question training texts for each first target answer training text respectively can be stored according to a preset data structure. The preset data structure refers to a storage format predefined by the system for efficiently organizing and managing data. In this way, these key data can be stored and managed systematically to ensure the efficient progress of subsequent operations.

[0090] In a specific example, multiple question training texts are input into the system , and the system determines the first target answer training texts and their corresponding target causal evaluation values . For the convenience of management, the system stores these data according to a preset data structure. For example, the stored data is constructed in the form of triples: In this way, the system can efficiently store and manage this data, providing an orderly basis for subsequent model training and data analysis.

[0091] Step 403: Input the question training text into the lightweight large language model to obtain the relevance score of the question training text to the second target answer training text among multiple answer training texts under the lightweight large language model.

[0092] In some embodiments of the present application, in order to evaluate the performance of the lightweight large language model when processing the question training text and generate the relevance scores of multiple answer training texts, the question training text will be input into the lightweight large language model to obtain the relevance score of the question training text to the second target answer training text among multiple answer training texts under the lightweight large language model. The relevance score is used to measure the matching degree between the answer training text and the question training text. In this way, the performance of the lightweight large language model can be evaluated, and data support can be provided for subsequent model optimization.

[0093] In a specific example, the experimenter inputs the question training text "What is an embedded model?" into the lightweight large language model. The model analyzes multiple answer training texts and generates the relevance score for each answer text. For example, the relevance score of the answer training text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" is 0.85. In this way, the system can evaluate the performance of the lightweight large language model when processing the question training text and generate accurate relevance scores, providing a reference for subsequent model optimization.

[0094] Step 404: Establish a training loss function for the lightweight large language model according to the relevance score corresponding to each question training text and the target causal evaluation value.

[0095] In some embodiments of the present application, in order to utilize the relevance score of the question training text and the target causal evaluation value to optimize the performance of the lightweight large language model, a training loss function for the lightweight large language model will be established according to the relevance score corresponding to each question training text and the target causal evaluation value. The training loss function is used to measure the error between the model prediction result and the true causal evaluation value, and the accuracy and robustness of the model are improved by minimizing this error. In this way, the model parameters can be precisely adjusted for the question training text and the target answer training text, thereby improving the inference and generation ability of the model.

[0096] In a specific example, the question training text "What is an embedded model?" can be input into the lightweight large language model, and multiple relevance scores for the answer training texts can be obtained. For example, the relevance score of the answer training text "An embedded model is a lightweight neural network model that runs in an environment with limited computing resources" is 0.85. Then, the system combines this relevance score with the previously calculated target causal evaluation value to establish a training loss function. This loss function is used to measure the error between the relevance score and the target causal evaluation value, and the model parameters are optimized by minimizing the error. Finally, the system will obtain an optimized lightweight large language model that can efficiently process natural language tasks in a low-computing-resource environment.

[0097] Optionally, in some embodiments of the present application, during the process of optimizing the Bidirectional Encoder Representations from Transformers (BERT) model in the manner of a regression task, the training objective can be determined as minimizing the relevance score of the BERT prediction with the error between the true CIS score. At this time, the loss function adopts the mean squared error (MSE) loss, so the training loss function can refer to the following formula: , where is the mean squared error loss function, is the predicted by BERT and the relevance score between them. In this way, while maintaining the causal reasoning ability, the inference calculation cost can be significantly reduced, enabling the retrieval task to not only take into account efficiency but also ensure high-quality causal reasoning ability, thereby enhancing the practical application value of the information retrieval system.

[0098] Step 405, train the lightweight large language model through the training loss function to obtain a trained lightweight large language model.

[0099] In some embodiments of the present application, in order to enable the lightweight large language model to process natural language tasks more accurately by optimizing the model parameters, the previously established training loss function will be used to train the lightweight large language model. The training loss function is used to measure the error between the model prediction result and the true value. By minimizing this error, the performance of the model will be improved. After executing this step, the system will obtain an optimized and trained lightweight large language model with high inference and generation capabilities.

[0100] In a specific example, the previously established training loss function can be utilized to optimize the model parameters. The training loss function calculates the error between the relevance score of the question training text and multiple answer training texts and the target causal evaluation value. Through repeated training, the system gradually adjusts the model parameters to enable the lightweight large language model to generate answers more accurately.

[0101] In summary, in the embodiments of the present application, by determining the causal evaluation value of the question text for each answer text, it not only relies on semantic similarity but also can quantify the actual helpfulness of the answer text to the question text, thereby avoiding misleading information. Furthermore, according to the causal evaluation value, the target answer text corresponding to the maximum causal evaluation value is selected to ensure that the most relevant text is given priority and then selected, thus significantly improving the accuracy of generating answers. Therefore, based on the method of the embodiments of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it surpasses traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of generating answers. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiments of the present application, the causal reasoning ability is transferred to the lightweight model through the knowledge transfer method of the present application, solving the high computational cost problem in the prior art and enabling the method to still have the ability to operate efficiently in a low-resource environment.

[0102] As Figure 5 shown, the embodiments of the present application also disclose a natural language processing device 50 for a large language model based on causal knowledge retrieval, including: A splicing module 501, configured to splice the question text with multiple answer texts to be matched respectively, so as to obtain question-answer texts corresponding to each answer text under the question text; An evaluation module 502, configured to determine the causal evaluation value of the question text for each answer text according to the corresponding question-answer text and answer text; the causal evaluation value is positively correlated with the conditional probability of content matching between the answer text and the question text when the answer text is the feedback result of the question text; A selection module 503, configured to determine the answer text corresponding to the maximum causal evaluation value among the multiple causal evaluation values as the target answer text matching the question text.

[0103] Optionally, the evaluation module 502 includes: A probability data sub-module, configured to determine the question-answer conditional probability data of content matching between the question-answer text and the question text when the answer text is the feedback result of the question text, and the answer probability data that the answer text is the feedback result of the question text; An evaluation value sub-module, which is used to determine the causal evaluation value of the answer text based on the logarithm of the ratio of the Q&A probability value and the answer probability value determined from the Q&A conditional probability data and the answer probability data; the Q&A probability value is the probability value of the conditional probability of the content matching between the Q&A text and the question text when the answer text is the feedback result of the question text; the answer probability value is the probability value of the probability that the answer text is the feedback result of the question text.

[0104] Optionally, the probability data sub-module includes: A word segmentation sequence unit, which is used to determine the answer word segmentation sequence of the answer text and the Q&A word segmentation sequence of the Q&A text according to the word segmentation extraction of the answer text; the answer sequence items of the answer word segmentation sequence are composed of each sequence word in the answer text recorded in order; the Q&A sequence items of the Q&A word segmentation sequence are composed of each sequence word in the answer text recorded in order and the question Q&A sequence items composed of the question text; the question Q&A sequence items are the first Q&A sequence items in the Q&A sequence. A previous item sequence unit, which is used to determine the answer previous item sequence of the answer text and the Q&A previous item sequence of the Q&A text respectively according to the extended selection starting from the first item of the answer word segmentation sequence and the Q&A word segmentation sequence; the sequence items of the answer previous item sequence and the Q&A previous item sequence correspond to the sequence items of the Q&A word segmentation sequence in order one by one. A component unit, which is used to record the conditional probability of each sequence item of the Q&A word segmentation sequence under the sequence item of the corresponding Q&A previous item sequence of the Q&A word segmentation sequence as a component value of the Q&A conditional probability data respectively, and record the conditional probability of each sequence item of the answer word segmentation sequence under the sequence item of the corresponding answer previous item sequence of the answer word segmentation sequence as a component value of the answer probability data respectively.

[0105] Optionally, the evaluation value sub-module includes: A logarithm unit, which is used to perform logarithmic operations on each component value of the Q&A conditional probability data and the answer probability data respectively to obtain the Q&A scoring data of the Q&A text and the answer scoring data of the answer text; An evaluation unit, which is used to determine the difference between the sum of each component value of the Q&A scoring data and the sum of each component value of the answer scoring data as the causal evaluation value.

[0106] Optionally, the large language model natural language processing device 50 based on causal knowledge retrieval further includes: A preliminary screening module, which is used to perform similarity matching between the question text and multiple texts according to the preset similarity conditions to obtain multiple answer texts to be matched.

[0107] As Figure 6 shown, the embodiment of the present application also discloses a knowledge transfer device 60 of a large language model based on causal knowledge retrieval, including: A training set module 601 for obtaining multiple question training texts; A transfer set module 602 for respectively inputting the multiple question training texts into a large language model based on causal knowledge retrieval to determine, from multiple answer training texts, first target answer training texts respectively corresponding to each question training text, and target causal evaluation values of the question training texts for each first target answer training text; the target causal evaluation value is used to characterize the conditional probability of the content matching between the first target answer training text and the question training text in the case where the first target answer training text is a feedback result of the question training text; A transfer training module 603 for training a lightweight large language model according to multiple groups of corresponding question training texts, first target answer training texts, and target causal evaluation values to obtain a trained lightweight large language model.

[0108] Optionally, the transfer training module 603 includes: A scoring sub-module for inputting a question training text into the lightweight large language model to obtain a relevance score of the question training text for a second target answer training text among multiple answer training texts under the lightweight large language model; A loss sub-module for establishing a training loss function for the lightweight large language model according to the relevance scores respectively corresponding to each question training text and the target causal evaluation values; A training sub-module for training the lightweight large language model through the training loss function to obtain a trained lightweight large language model.

[0109] In summary, in the embodiment of the present application, by determining the causal evaluation value of a question text for each answer text, it not only relies on semantic similarity but also can quantify the actual helpfulness of the answer text to the question text, thereby avoiding misleading information; and then, according to the causal evaluation value, the target answer text corresponding to the maximum causal evaluation value is selected to ensure that the most relevant text is given priority and then selected, thus significantly improving the accuracy of generating answers. Therefore, based on the method of the embodiment of the present application, by quantifying the actual contribution of the answer text through the causal evaluation value, it surpasses the traditional semantic similarity retrieval, fundamentally solves the problem of misleading answers, and improves the accuracy of generating answers. It solves the problem of inaccurate inference results in the process of natural language processing. In addition, in the embodiment of the present application, by using the knowledge transfer method of the present application to transfer the causal reasoning ability to a lightweight model, it solves the problem of high computational cost in the prior art, enabling the method to still have the ability to operate efficiently in a low-resource environment.

[0110] The embodiments of the present application also provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements each process of the above-mentioned embodiments of the natural language processing method and knowledge transfer method of the large language model based on causal knowledge retrieval, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0111] Figure 7 FIG. 4 is a block diagram of an electronic device 700 provided by an embodiment of the present application. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0112] Referring Figure 7 to FIG. 4, the electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0113] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned natural language processing method and knowledge transfer method of the large language model based on causal knowledge retrieval. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0114] The memory 704 is used to store various types of data to support the operation of the electronic device 700. Examples of these data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, multimedia, etc. The memory 704 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0115] The power supply component 706 provides power for various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 700.

[0116] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0117] The audio component 710 is used to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC) that is used to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 further includes a speaker for outputting audio signals.

[0118] The I / O interface 712 provides an interface between the processing component 702 and a peripheral interface module, and the peripheral interface module may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power-on button, and a lock button.

[0119] The sensor assembly 714 includes one or more sensors for providing a status assessment of various aspects of the electronic device 700. For example, the sensor assembly 714 can detect the on / off state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect a change in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and a change in the temperature of the electronic device 700. The sensor assembly 714 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0120] The communication component 716 is used to facilitate communication between the electronic device 700 and other devices in a wired or wireless manner. The electronic device 700 can access a wireless network based on communication standards, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 7G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0121] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for implementing the large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval provided in the embodiments of the present application.

[0122] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the above instructions can be executed by a processor 720 of the electronic device 700 to complete the above-mentioned large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0123] Figure 8 is a block diagram of an electronic device 800 shown according to an exemplary embodiment. For example, the electronic device 800 may be provided as a server. Referring to Figure 8 , the electronic device 800 includes a processing component 822, which further includes one or more processors, and memory resources represented by a memory 832 for storing instructions executable by the processing component 822, such as application programs. The application programs stored in the memory 832 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval provided by the embodiments of the present application.

[0124] The electronic device 800 may also include a power supply component 826 configured to perform power management of the electronic device 800, a wired or wireless network interface 850 configured to connect the electronic device 800 to a network, and an input / output (I / O) interface 858. The electronic device 800 may operate based on an operating system stored in the memory 832, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSD TM or the like.

[0125] The embodiments of the present application also provide a computer program product, including a computer program, where the computer program, when executed by a processor, implements the large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval.

[0126] Those skilled in the art will readily think of other implementations of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0127] It should be understood that the present application is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

[0128] Each embodiment in this specification is described in a progressive manner, with the key point of each embodiment being the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0129] It is easy for those skilled in the art to think that any combination application of the above-mentioned embodiments is feasible. Therefore, any combination of the above-mentioned embodiments is an implementation scheme of this application. However, due to space limitations, this manual will not elaborate on each one here.

[0130] The natural language processing method and knowledge transfer method of the large language model based on causal knowledge retrieval provided here are not inherently related to any specific computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based here. According to the above description, it is obvious to construct the required structure of a system with the solution of this application. In addition, this application is not directed to any specific programming language. It should be understood that the content of this application described here can be implemented using various programming languages, and the description of a specific language above is to disclose the best implementation mode of this application.

[0131] In the specification provided here, a large number of specific details are described. However, it can be understood that the embodiments of this application can be practiced without these specific details. In some instances, well-known methods, structures and technologies are not shown in detail so as not to obscure the understanding of this specification.

[0132] Similarly, it should be understood that, in order to streamline this application and help understand one or more of the various aspects of the application, in the above description of the exemplary embodiments of this application, the various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected by the claims, the aspects of the application lie in less than all the features of the single embodiments disclosed above. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim stands on its own as a separate embodiment of this application.

[0133] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from that embodiment. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted for all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0134] In addition, those skilled in the art can understand that although some of the embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means that it is within the scope of this application and forms different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0135] Each component embodiment of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval according to the embodiments of the present application. The present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0136] In yet another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when run on a computer, causes the computer to execute the large language model natural language processing method and knowledge transfer method based on causal knowledge retrieval according to the embodiments of the present application.

[0137] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0138] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names.

[0139] It should be noted that for the method embodiments of the present application, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0140] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of a system or device, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A natural language processing method for large language models based on causal knowledge retrieval, characterized in that, Including: Splicing the question text with multiple answer texts to be matched respectively to obtain question-answer texts respectively corresponding to each of the answer texts under the question text; Determining the causal evaluation value of the question text for each of the answer texts according to the corresponding question-answer text and answer text; the causal evaluation value is positively correlated with the conditional probability of the content matching between the answer text and the question text when the answer text is the feedback result of the question text; Determining the answer text corresponding to the largest causal evaluation value among the multiple causal evaluation values as the target answer text matching the question text.

2. The natural language processing method for large language models based on causal knowledge retrieval according to claim 1, wherein, The determining the causal evaluation value of the question text for each of the answer texts according to the corresponding question-answer text and answer text includes: Determining the question-answer conditional probability data of the content matching between the question-answer text and the question text when the answer text is the feedback result of the question text, and the answer probability data that the answer text is the feedback result of the question text; Determining the logarithm of the ratio of the question-answer probability value to the answer probability value determined according to the question-answer conditional probability data and the answer probability data as the causal evaluation value of the answer text; the question-answer probability value is the probability value of the conditional probability of the content matching between the question-answer text and the question text when the answer text is the feedback result of the question text; the answer probability value is the probability value of the probability that the answer text is the feedback result of the question text.

3. The natural language processing method for large language models based on causal knowledge retrieval according to claim 2, wherein, The determining the question-answer conditional probability data of the content matching between the question-answer text and the question text when the answer text is the feedback result of the question text, and the answer probability data that the answer text is the feedback result of the question text includes: Determining the answer token sequence of the answer text and the question-answer token sequence of the question-answer text according to the token extraction of the answer text; the answer sequence items of the answer token sequence are composed of each sequence token recorded in order in the answer text; the question-answer sequence items of the question-answer token sequence are composed of each sequence token recorded in order in the answer text and the question-answer sequence item composed of the question text; the question-answer sequence item is the first question-answer sequence item in the question-answer sequence; Respectively determining the answer previous item sequence of the answer text and the question-answer previous item sequence of the question-answer text according to the extended selection starting from the first item of the answer token sequence and the question-answer token sequence; the sequence items of the answer previous item sequence and the question-answer previous item sequence correspond to the sequence items of the question-answer token sequence in order one by one; Recording the conditional probability of each sequence item of the question-answer token sequence under the sequence item of the corresponding question-answer previous item sequence as a component value of the question-answer conditional probability data, and recording the conditional probability of each sequence item of the answer token sequence under the sequence item of the corresponding answer previous item sequence as a component value of the answer probability data.

4. The natural language processing method for large language models based on causal knowledge retrieval according to claim 2, characterized in that, The question-and-answer conditional probability data and the answer probability data respectively include multiple component values. Determining the logarithm of the ratio of the question-and-answer probability value and the answer probability value determined according to the question-and-answer conditional probability data and the answer probability data as the causal evaluation value of the answer text includes: Performing a logarithm operation on each component value of the question-and-answer conditional probability data and the answer probability data to obtain the question-and-answer scoring data of the question-and-answer text and the answer scoring data of the answer text; Determining the difference between the sum of each component value of the question-and-answer scoring data and the sum of each component value of the answer scoring data as the causal evaluation value.

5. The natural language processing method for large language models based on causal knowledge retrieval according to claim 1, wherein The method further includes: According to a preset similarity condition, performing similarity matching between the question text and multiple texts to obtain multiple answer texts to be matched.

6. A knowledge transfer method for a large language model based on causal knowledge retrieval, characterized in that, Including: Obtaining multiple question training texts; Inputting the multiple question training texts into a large language model based on causal knowledge retrieval respectively to determine a first target answer training text corresponding to each question training text from multiple answer training texts, and the target causal evaluation value of each question training text for each first target answer training text; The target causal evaluation value is used to represent the conditional probability of the content matching between the first target answer training text and the question training text in the case that the first target answer training text is the feedback result of the question training text; Training a lightweight large language model according to multiple groups of corresponding question training texts, first target answer training texts, and target causal evaluation values to obtain the trained lightweight large language model.

7. The knowledge transfer method of the large language model based on causal knowledge retrieval according to claim 6, characterized in that, The training of the lightweight large language model according to multiple groups of corresponding question training texts, first target answer training texts, and target causal evaluation values to obtain the trained lightweight large language model includes: Inputting the question training text into the lightweight large language model to obtain the relevance score of the question training text for a second target answer training text among multiple answer training texts under the lightweight large language model; Establishing a training loss function for the lightweight large language model according to the relevance score corresponding to each question training text and the target causal evaluation value; Training the lightweight large language model through the training loss function to obtain the trained lightweight large language model.

8. A natural language processing device for large language models based on causal knowledge retrieval, characterized in that, Including: A splicing module for splicing the question text and multiple answer texts to be matched respectively to obtain question-and-answer texts corresponding to each answer text under the question text; An evaluation module for determining the causal evaluation value of the question text for each answer text according to the corresponding question-and-answer text and answer text; the causal evaluation value is positively correlated with the conditional probability of the content matching between the answer text and the question text in the case that the answer text is the feedback result of the question text. A selection module, configured to determine, as a target answer text matching the question text, the answer text corresponding to the largest causal evaluation value among the multiple causal evaluation values.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images