Knowledge updating method, device and equipment of model based on industrial internet, medium and program product

Abstract

The embodiment of the invention provides a knowledge updating method and device of a model based on the industrial internet, equipment, a medium and a program product, and relates to the technical field of big data. The knowledge updating method of the model based on the industrial internet comprises the steps of obtaining a plurality of question and answer pairs from a plurality of knowledge documents in response to an adjustment instruction, determining a confusion value of a target model for each question and answer pair, determining a knowledge type corresponding to each question and answer pair based on the confusion value corresponding to each question and answer pair, and when the knowledge type indicates that the question and answer pair is extraterritorial knowledge, updating the knowledge of the model based on the industrial internet. Determining to-be-adjusted neurons in the target model based on the question and answer pair, and performing parameter replacement on the to-be-adjusted neurons to realize knowledge updating of the target model; according to the method, the question and answer pairs of which the knowledge types are out-of-domain knowledge are screened, the knowledge updating burden of the target model is reduced, the target parameters are replaced with the adjustment parameters, new knowledge is injected into the target model, resource consumption is reduced, and the knowledge updating efficiency of the model is improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a knowledge update method, apparatus, equipment, medium and program product based on an industrial internet model. Background Technology

[0002] Large Language Models (LLMs) are deep learning-based natural language processing models that are trained on large-scale text data to understand and generate natural language text. Although LLMs are trained on vast amounts of text data, the knowledge within the model is not always up-to-date over time, necessitating model updates to ensure the timeliness of its knowledge.

[0003] In existing technologies, the fine-tuning update method is usually used to update the knowledge of the model. It introduces a new dataset containing new knowledge and uses the new dataset to further train the existing model to adapt it to the new knowledge content, thereby supplementing or correcting the original knowledge in the model.

[0004] However, updating the model's knowledge using the above update method requires a large amount of computing resources, resulting in low update efficiency. Summary of the Invention

[0005] This application provides a knowledge update method, apparatus, device, medium, and program product based on an industrial internet model, to solve the problem that existing knowledge update methods for models require a lot of computing resources, resulting in low update efficiency.

[0006] In a first aspect, embodiments of this application provide a knowledge update method based on an industrial internet model, comprising:

[0007] In response to adjustment instructions, multiple question-answer pairs are retrieved from multiple knowledge documents;

[0008] Determine the perplexity value of the target model for each of the aforementioned question-answer pairs;

[0009] Based on the confusion value corresponding to each question-answer pair, the knowledge type corresponding to each question-answer pair is determined;

[0010] When the knowledge type indicates that the question-answer pair is external knowledge, the neurons to be adjusted in the target model are determined based on the question-answer pair, and the parameters of the neurons to be adjusted are replaced to achieve knowledge update of the target model.

[0011] Optionally, obtaining multiple question-answer pairs from multiple knowledge documents includes:

[0012] The knowledge document is input into a pre-trained large language model to obtain multiple question-answer pairs.

[0013] Optionally, determining the perplexity value of the target model for each question-answer pair includes:

[0014] For each question-answer pair, the question-answer pair is split into multiple consecutive word units;

[0015] Input the lexical units into the target model to predict the lexical units that follow each of the lexical units;

[0016] Based on the continuity relationship between each lexical unit, the predicted lexical unit corresponding to each lexical unit, and the total number of lexical units, the perplexity value of each question-answer pair is determined.

[0017] Optionally, the multiple knowledge types correspond one-to-one with multiple pre-set confusion value threshold ranges;

[0018] The step of determining the knowledge type corresponding to each question-answer pair based on the confusion value of each question-answer pair includes:

[0019] From the multiple perplexity threshold ranges, the target perplexity threshold range corresponding to the perplexity value of each question-answer pair is matched;

[0020] Based on the target confusion value threshold range corresponding to each question-answer pair, the knowledge type corresponding to each question-answer pair is determined.

[0021] Optionally, the method further includes:

[0022] When the knowledge type indicates that the question-answer pair is outdated or incorrect knowledge, the question-answer pair is deleted.

[0023] Optionally, the step of determining the neurons to be adjusted in the target model based on the question-answer pair and replacing the parameters of the neurons to be adjusted to achieve knowledge updating of the target model includes:

[0024] The target word is determined from the word elements contained in the question-answer pair, and the neuron corresponding to the target word is located in any layer of the neural network of the target model as the neuron to be adjusted.

[0025] Causal intervention analysis was used to determine the target parameters in the neurons to be adjusted that are related to the target word.

[0026] Based on the target parameters, the adjustment parameters are calculated using the least squares method, and the target parameters are replaced with the adjustment parameters to achieve knowledge update of the target model.

[0027] Secondly, embodiments of this application provide a knowledge update device based on an industrial internet model, comprising:

[0028] The knowledge acquisition module is used to retrieve multiple question-answer pairs from multiple knowledge documents in response to adjustment commands;

[0029] The determination module is used to determine the perplexity value of the target model for each of the question-answer pairs;

[0030] The knowledge classification module is used to determine the knowledge type corresponding to each question-answer pair based on the confusion value corresponding to each question-answer pair;

[0031] The knowledge injection module is used to determine the neurons to be adjusted in the target model based on the question-answer pair when the knowledge type indicates that the question-answer pair is out-of-domain knowledge, and to replace the parameters of the neurons to be adjusted in order to realize the knowledge update of the target model.

[0032] Optionally, the knowledge acquisition module is further configured to input the knowledge document into a pre-trained large language model to obtain multiple question-answer pairs.

[0033] Optionally, the determining module is further configured to split each question-answer pair into multiple consecutive word units;

[0034] The determining module is further configured to input the word elements into the target model and predict the predicted word elements that follow each of the word elements;

[0035] The determining module is further configured to determine the perplexity value of each question-answer pair based on the continuous relationship between each word, the predicted word corresponding to each word, and the total number of words.

[0036] Optionally, the multiple knowledge types correspond one-to-one with multiple pre-set confusion value threshold ranges;

[0037] The knowledge classification module is also used to match the target confusion value threshold range corresponding to the confusion value of each question-answer pair from multiple confusion value threshold ranges;

[0038] The knowledge classification module is also used to determine the knowledge type corresponding to each question-answer pair based on the target confusion value threshold range corresponding to each question-answer pair.

[0039] Optionally, the device further includes: a deletion module;

[0040] The deletion module is used to delete the question-answer pair when the knowledge type indicates that the question-answer pair is outdated or incorrect knowledge.

[0041] Optionally, the knowledge injection module is further configured to determine the target word from the word elements contained in the question-answer pair, and locate the neuron corresponding to the target word from any layer of the neural network of the target model as the neuron to be adjusted;

[0042] The knowledge injection module is also used to determine the target parameters related to the target word in the neuron to be adjusted by using causal intervention analysis;

[0043] The knowledge injection module is further configured to calculate adjustment parameters based on the target parameters using the least squares method, and replace the target parameters with the adjustment parameters to achieve knowledge update of the target model.

[0044] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory, causing the processor to perform the knowledge update method based on the Industrial Internet model as described in the first aspect and / or various possible embodiments of the first aspect.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the knowledge update method based on the Industrial Internet model as described in the first aspect and / or various possible implementations of the first aspect.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the knowledge update method based on the Industrial Internet model as described in the first aspect and / or various possible implementations of the first aspect.

[0049] The knowledge update method for industrial internet-based models provided in this application, in response to adjustment instructions, obtains multiple question-answer pairs from multiple knowledge documents, determines the perplexity value of the target model for each question-answer pair, determines the knowledge type corresponding to each question-answer pair based on the perplexity value, and when the knowledge type indicates that the question-answer pair is out-of-domain knowledge, determines the neurons to be adjusted in the target model based on the question-answer pair, and replaces the parameters of the neurons to be adjusted to achieve knowledge update of the target model. This method reduces the knowledge update burden of the target model by screening question-answer pairs with knowledge types that are out-of-domain knowledge, and injects new knowledge into the target model by replacing the target parameters with adjustment parameters, thereby reducing resource consumption and improving the efficiency of knowledge update of the model. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] Figure 1 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 1 ;

[0052] Figure 2 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 2 ;

[0053] Figure 3 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 3 ;

[0054] Figure 4 A schematic diagram illustrating the knowledge classification of question-answer pairs provided in an embodiment of this application;

[0055] Figure 5 A schematic diagram illustrating extraterritorial knowledge provided in an embodiment of this application;

[0056] Figure 6 A schematic diagram of the structure of a knowledge update device based on an industrial internet model provided in this application embodiment;

[0057] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0061] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] First, a brief introduction to the technical terms used in the embodiments of this application:

[0063] Large Language Model (LLM): refers to a deep learning model trained on a large amount of text data that can generate natural language text or understand the meaning of language text.

[0064] Knowledge editing: Knowledge editing is a method of updating an LLM by avoiding the computational burden associated with training a completely new model. Its goal is to change the behavior of the LLM in a specific domain without affecting the performance of other inputs.

[0065] Question-answer pairs are paired data units consisting of a question and a corresponding answer, and are a basic data form in the field of natural language processing.

[0066] Perplexity (PPL) measures a language model's ability to predict the probability of a given text, specifically its ability to predict the next word given previous words. PPL reflects the degree of "confusion" the model experiences with the language data; a lower PPL indicates a more accurate and fluent understanding of the text, while a higher PPL indicates greater difficulty in understanding or generating the text.

[0067] Extraterritorial knowledge refers to information that extends beyond the topics or domains covered by the model's training data.

[0068] Incorrect knowledge refers to factual errors or misleading information contained in the content generated by the model.

[0069] Outdated knowledge refers to information that was once correct but is no longer accurate in the current context.

[0070] As mentioned earlier, large language models learn patterns from massive amounts of text data to understand and generate natural language. These models can support a variety of tasks, including automatic question answering, machine translation, text summarization, dialogue systems, and content creation.

[0071] Existing model knowledge update methods, when using fine-tuning to update knowledge, require retraining the model with a dataset containing new knowledge. Even if the size of the new data is smaller than the original pre-training data, it is still necessary to traverse the entire dataset multiple times and update all or most of the parameters, which means that it consumes a lot of computing resources, the training time is long, and thus the update efficiency is low.

[0072] In view of this, this application proposes a knowledge update method for industrial internet-based models. First, knowledge represented in question-and-answer pairs is extracted from multiple continuously updated knowledge documents, thus improving the accuracy of knowledge representation. Then, the perplexity value of the target model for each question-and-answer pair is calculated. Based on the magnitude of the perplexity value, the knowledge type corresponding to each question-and-answer pair is determined. Using the perplexity value as a screening criterion allows for more precise knowledge filtering, avoiding the repeated injection of outdated or erroneous knowledge. Finally, knowledge editing is performed on question-and-answer pairs whose knowledge type is out-of-domain knowledge to obtain adjustment parameters. These adjustment parameters are then used to replace the original parameters of the neurons to be adjusted in the target model, thereby achieving knowledge update of the target model. This method can update the internal parameters of the model without retraining it, enabling it to acquire new knowledge, reducing resource consumption, and improving the efficiency of knowledge update.

[0073] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0074] Figure 1 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 1 Taking a knowledge update device based on an industrial internet model as an example, such as... Figure 1 As shown, the knowledge update method based on the Industrial Internet model provided in this embodiment includes the following steps:

[0075] S101. In response to the adjustment instruction, retrieve multiple question-answer pairs from multiple knowledge documents.

[0076] In this embodiment of the application, the multiple knowledge documents can be continuously updated data sources such as news, knowledge documents, and newspapers. When the system detects that knowledge needs to be updated, it will issue an adjustment instruction. After receiving the adjustment instruction, the knowledge acquisition module will start to extract multiple question-answer pairs from the multiple knowledge documents.

[0077] The adjustment command can be triggered based on a time period, for example, this step can be triggered once every preset time interval (such as every 12 hours / day).

[0078] Understandably, question-answer pairs can accommodate longer contexts, more complex grammatical and semantic information, and represent knowledge more accurately. Secondly, question-answer pairs are more similar to the input-output format of large language models, making them more suitable for updating the model's knowledge.

[0079] In some embodiments, pattern matching is performed on the content of knowledge documents based on preset question templates to identify key information and generate corresponding questions. Then, by performing syntactic analysis on natural language sentences, declarative sentences are transformed into interrogative sentences. Based on document structural features (such as titles, lists, and tables), question-and-answer rules are constructed to achieve automatic association and matching of questions and answers (such as using chapter titles as questions and the following paragraphs as answers).

[0080] S102. Determine the perplexity value of the target model for each question-answer pair.

[0081] The target model can be a large language model to be updated, and the perplexity value can be a quantification of the uncertainty or prediction difficulty of the target model for question-answer pairs.

[0082] Understandably, perplexity values ​​are used to reflect the accuracy and confidence of the target model when predicting unknown information (such as the answer) based on known information (such as the question part).

[0083] For example, each question-answer pair can be input into the target model, and the target model can be used to reason about each question-answer pair by calculating the predicted probability of each word to determine the perplexity value of each question-answer pair.

[0084] S103. Based on the confusion value corresponding to each question-answer pair, determine the knowledge type corresponding to each question-answer pair.

[0085] In this embodiment of the application, the knowledge type corresponding to each question-and-answer pair may include: outdated knowledge, erroneous knowledge, foreign knowledge, etc.

[0086] It's understandable that different types of questions and answers will have different levels of confusion.

[0087] If the perplexity value of a question-and-answer pair is high, it indicates that the target model has low confidence and accuracy for this specific question and answer. This may be because the information contained in the answer was once factual but has since changed. Therefore, the knowledge type of this question-and-answer pair is outdated knowledge. If the perplexity value of a question-and-answer pair is very high, it may be because the obviously wrong answer makes the target model feel "contradictory" and difficult to predict. In this case, the knowledge type of this question-and-answer pair is incorrect knowledge. If the perplexity value of a question-and-answer pair is extremely high, it indicates that the target model encountered great difficulty in trying to predict the answer to this question-and-answer pair. The target model may have never seen or lack relevant data. In other words, the knowledge type of this question-and-answer pair is out-of-domain knowledge.

[0088] S104. When the knowledge type indicates that the question-answer pair is external knowledge, determine the neurons to be adjusted in the target model based on the question-answer pair, and replace the parameters of the neurons to be adjusted.

[0089] To avoid redundant knowledge injection and reduce the workload of knowledge fusion in the target model, it is necessary to screen question-answer pairs of different knowledge types to obtain the knowledge that the target model finds most confusing, i.e., external knowledge, and use this external knowledge to update the target model.

[0090] Figure 2 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 2 The following will refer to Figure 2 The knowledge update process in the embodiments of this application is described as follows:

[0091] First, multiple knowledge documents, such as Knowledge Document 1, Knowledge Document 2, ..., Knowledge Document n, provide a continuously updated data source for the target model. During the knowledge acquisition phase, knowledge is extracted from these documents to obtain multiple question-answer pairs. Then, in the knowledge classification phase, the target model is used to determine the perplexity value corresponding to each question-answer pair. Based on these perplexity values, the knowledge type of each question-answer pair is classified, thereby filtering out the out-of-domain knowledge used to update the target model—i.e., new knowledge. Finally, in the knowledge injection phase, new knowledge is determined from the question-answer pairs of this out-of-domain knowledge type. Using knowledge editing techniques, the location of the neuron to be adjusted in the target model is located, and the new parameter values ​​required to update the knowledge at that neuron are calculated. These new parameters are then injected into the target model through parameter replacement, thus achieving knowledge updating of the target model.

[0092] In this way, compared with the existing fine-tuning methods, the internal parameters of the target model can be updated to enable it to learn new knowledge without retraining the target model. Moreover, the edited model does not require the addition of new modules, which effectively reduces the inference burden of the updated target model.

[0093] The knowledge update method for industrial internet-based models provided in this application, in response to adjustment instructions, obtains multiple question-answer pairs from multiple knowledge documents, determines the perplexity value of the target model for each question-answer pair, determines the knowledge type corresponding to each question-answer pair based on the perplexity value, and when the knowledge type indicates that the question-answer pair is out-of-domain knowledge, determines the neurons to be adjusted in the target model based on the question-answer pair, and replaces the parameters of the neurons to be adjusted to achieve knowledge update of the target model. This method reduces the knowledge update burden of the target model by screening question-answer pairs with knowledge types that are out-of-domain knowledge, and injects new knowledge into the target model by replacing the target parameters with adjustment parameters, thereby reducing resource consumption and improving the efficiency of knowledge update of the model.

[0094] Figure 3 A flowchart illustrating the knowledge update method based on the Industrial Internet model provided in this application embodiment. Figure 3 This embodiment is in Figure 1 Based on the examples, the knowledge update method for the Industrial Internet-based model is described in detail. For example... Figure 3 As shown, the knowledge update method based on the Industrial Internet model provided in this embodiment includes:

[0095] S301. In response to the adjustment command, the knowledge document is input into the pre-trained large language model to obtain multiple question-answer pairs.

[0096] In this embodiment, prompting technology can be used to guide a large language model to output multiple question-answer pairs that include contextual expansion. For example, a suitable prompt template can be automatically matched according to the knowledge document type (such as "news" or "legal"). This prompt template is used to guide the large language model to extract question-answer pairs from the knowledge document. Then, a pre-trained large language model is called, and the content of the knowledge document and the prompt template are input into the large language model to obtain multiple question-answer pairs output by the large language model.

[0097] Thus, by utilizing prompting techniques, the prompting large language model generates knowledge representations with longer contexts in the form of question-answer pairs.

[0098] S302. For each question-answer pair, split the question-answer pair into multiple consecutive word units.

[0099] Optionally, before splitting the question-answer pair into multiple consecutive tokens, each question-answer pair can be preprocessed to convert it into a format that the target model can compute.

[0100] For example, a question-answer pair would be:

[0101] Q (Question): "What are the basic building blocks of matter?"

[0102] A (Answer): "Atom."

[0103] Convert the above question-and-answer pair into a form that the target model can calculate, such as: "What is the basic building unit of matter? Atom." Then split the above question-and-answer pair into tokens, such as: ["substance", "quality", "of", "basic", "composition", "unit", "is", "what", "?", "atom", "."].

[0104] S303. Input the tokens into the target model and predict the tokens following each token.

[0105] In this step, input the tokens split from the question-and-answer pair into the target model. The target model can predict the next token based on the previous tokens.

[0106] For example, based on "substance", predict the probability of "quality"; based on "substance quality", predict the probability of "of"; based on "substance quality of", predict the probability of "basic", until predicting the probability of the last token "."

[0107] S304. Determine the perplexity value of each question-and-answer pair based on the consecutive relationship between tokens, the predicted tokens corresponding to each token, and the total number of tokens.

[0108] In the embodiment of this application, first, through the target model, output the probability distribution of the next token at each position in the complete sentence corresponding to the question-and-answer pair, that is, the probability of all possible tokens. Here, it is necessary to focus on the probability of the real next token, that is, P("quality"|"substance"), P("of"|"substance quality"); then, based on the consecutive relationship between tokens, calculate the negative logarithm average of the predicted probabilities of all tokens in the answer part; finally, convert the result into a perplexity value through exponential operation.

[0109] For example, the perplexity value PPL can be calculated using the following formula:

[0110]

[0111] where N is the total number of tokens, is the probability of the real token predicted by the target model.

[0112] S305. Match the target perplexity value threshold range corresponding to the perplexity value of each question-and-answer pair from multiple perplexity value threshold ranges.

[0113] Among them, multiple knowledge types correspond to multiple preset perplexity value threshold ranges one by one.

[0114] The essence of this step is to classify the perplexity value corresponding to each question-and-answer pair into the corresponding interval in multiple preset perplexity value threshold ranges according to its numerical value.

[0115] For example, multiple perplexity threshold ranges are such as (20, 50], (50, 100], and (100, +∞), with each threshold range representing a different degree of perplexity of the target model for the question-answer pair.

[0116] S306. Based on the target confusion value threshold range corresponding to each question-answer pair, determine the knowledge type corresponding to each question-answer pair.

[0117] For example, if the perplexity value of a question-and-answer pair falls within the range of (20, 50), then the knowledge type corresponding to the question-and-answer pair is determined to be outdated knowledge; if the perplexity value of a question-and-answer pair falls within the range of (50, 100), then the knowledge type corresponding to the question-and-answer pair is determined to be incorrect knowledge; if the perplexity value of a question-and-answer pair falls within the range of (100, +∞), then the knowledge type corresponding to the question-and-answer pair is determined to be outside the domain.

[0118] Figure 4 This is a schematic diagram illustrating the knowledge classification of question-answer pairs provided in an embodiment of this application, such as... Figure 4 As shown, by calculating the perplexity value of the target model for each question-answer pair, the question-answer pairs corresponding to the knowledge document can be classified into: outside knowledge, erroneous knowledge, and outdated knowledge, thereby filtering out outside knowledge for knowledge updates of the target model.

[0119] Optionally, when the knowledge type indicates that the question-answer pair is outdated or incorrect knowledge, delete the question-answer pair.

[0120] In this way, new knowledge in knowledge documents can be effectively screened and filtered, avoiding the repeated injection of knowledge and improving the efficiency of knowledge updates.

[0121] S307. When the knowledge type indicates that the question-answer pair is external knowledge, the target word is determined from the word elements contained in the question-answer pair, and the neuron corresponding to the target word is located in any layer of the neural network of the target model as the neuron to be adjusted.

[0122] In this context, target lexical units can refer to keywords in the question-answer pair that result in high perplexity values ​​in the model.

[0123] For example, if the question-answer pair is determined to be out-of-domain knowledge based on the perplexity value of the target model, an attention mechanism can be used to identify the target word, extract the activation value of the target word in each layer, locate the target word in layer 28 where neurons #512 and #768 are activated, and then analyze the activation pattern and output influence of these two neurons to determine whether they are neurons to be adjusted.

[0124] Optionally, before knowledge injection, data augmentation of external knowledge can be performed using methods such as knowledge expansion and knowledge reversal to effectively improve the model editing effect.

[0125] This is due to the "curse of reversal" phenomenon in current large language models, where the model exhibits unexpectedly high accuracy on unseen data in certain situations. During model editing, when learning new knowledge, the model sometimes incorrectly rejects old knowledge or creates non-existent connections and inductions. To correct this, diverse question-answer pair training samples can be created through expansion and reversal. This will help the model more accurately understand and distinguish different knowledge, improving the effectiveness and accuracy of model editing.

[0126] S308. Using causal intervention analysis, determine the target parameters related to the target word in the neurons to be adjusted.

[0127] Among them, causal intervention analysis refers to assessing the impact of changes in parameters in neurons to be adjusted on the final output by simulating intervention operations, thereby locating the target parameters.

[0128] In one possible implementation, question-answer pairs containing the target word are input into the target model, and the activation values ​​of each layer of the model are recorded, along with the final output of the model. The representation of the target word is modified at the word embedding layer, for example, by replacing it with word vectors of other known words, or setting the embedding vector at that position to zero, thus simulating an external intervention condition. The model is run again to obtain the activation values ​​and output results of each layer after the intervention. Then, the changes in activation values ​​before and after the intervention are compared to identify the set of neurons most affected by the target word. The gradient of the activation tensor of the layer containing the neuron to be adjusted relative to the final loss function is calculated to assess its impact on the model output.

[0129] Finally, based on the gradient information corresponding to the neurons to be adjusted, the specific parameters connected to these neurons (such as rows / columns in the weight matrix, bias terms, etc.) are further located as target parameters for subsequent knowledge editing.

[0130] Figure 5 This is a schematic diagram illustrating the knowledge editing of extraterritorial knowledge provided in the embodiments of this application, such as... Figure 5 As shown, the input value first undergoes a linear transformation through the first-layer weight matrix W_1. The output of the first layer passes through a non-linear activation function to increase the model's expressive power. From the output after non-linear activation, "key values" related to external knowledge are identified. These "key values" are then passed as input to the second-layer weight matrix W_2 for further linear transformation. The output of the second layer passes through an activation function again, ultimately yielding a "new value," which is the model's prediction result. Based on the causal localization results, if certain "key values" are found to have a significant impact on the output, the target model can be updated by adjusting the parameters of the second-layer weight matrix.

[0131] S309. Based on the target parameter, calculate the adjustment parameter using the least squares method, and replace the target parameter with the adjustment parameter.

[0132] In this step, the least squares method can be used to construct an optimization objective function and find new parameters that minimize the prediction error, i.e., adjust the parameters.

[0133] For example, the objective function can be expressed using the following formula:

[0134]

[0135] in, Let X be the low-rank adjustment matrix, W be the matrix corresponding to the target parameters, X be the input data, and Y be the desired output.

[0136] If only the column Wt containing the target parameter in the target parameter matrix needs to be adjusted, the above formula can be simplified to the following formula (3):

[0137]

[0138] in, These are the input features for the activation target parameters. That is the expected output.

[0139] Then, by solving the above formula (3), the solution to the formula is obtained as follows: Therefore, the adjustment parameters are determined as follows: .

[0140] Finally, use the adjustment parameters. Replace the target column in the matrix corresponding to the target parameter. .

[0141] Thus, by using the Rank-One Model Editing (ROME) method, knowledge editing can be achieved by only changing the local structure of the matrix, without affecting other irrelevant knowledge in the model.

[0142] The knowledge update method based on the Industrial Internet model provided in this application improves the accuracy of new knowledge representation by extracting knowledge in question-answer pairs from document content using a pre-trained large language model. In the knowledge classification stage, perplexity values ​​are used to classify each question-answer pair, achieving effective knowledge screening and filtering, avoiding repeated injection of new knowledge, and improving the efficiency of subsequent knowledge injection. In the knowledge injection stage, knowledge editing technology is used to inject new knowledge into the target model by adjusting parameters to replace target parameters, updating the target model's knowledge, reducing resource consumption, and improving the efficiency of knowledge update for the target model.

[0143] Figure 6 A schematic diagram of the structure of the knowledge update device based on the Industrial Internet model provided in this application is shown below. Figure 6 As shown, the knowledge update device 600 based on the Industrial Internet model provided in this embodiment includes:

[0144] The knowledge acquisition module 601 is used to retrieve multiple question-answer pairs from multiple knowledge documents in response to adjustment instructions;

[0145] The determination module 602 is used to determine the perplexity value of the target model for each of the question-answer pairs;

[0146] The knowledge classification module 603 is used to determine the knowledge type corresponding to each question-answer pair based on the confusion value corresponding to each question-answer pair;

[0147] The knowledge injection module 604 is used to determine the neurons to be adjusted in the target model based on the question-answer pair when the knowledge type indicates that the question-answer pair is external knowledge, and to replace the parameters of the neurons to be adjusted in order to realize the knowledge update of the target model.

[0148] Optionally, the knowledge acquisition module 601 is further configured to input the knowledge document into a pre-trained large language model to obtain multiple question-answer pairs.

[0149] Optionally, the determining module 602 is further configured to split each question-answer pair into multiple consecutive word units;

[0150] The determining module 602 is further configured to input the word elements into the target model and predict the predicted word elements that follow each of the word elements;

[0151] The determining module 602 is further configured to determine the perplexity value of each question-answer pair based on the continuous relationship between each word, the predicted word corresponding to each word, and the total number of words.

[0152] Optionally, the multiple knowledge types correspond one-to-one with multiple pre-set confusion value threshold ranges;

[0153] The knowledge classification module 603 is further configured to match the target confusion value threshold range corresponding to the confusion value of each question-answer pair from the multiple confusion value threshold ranges;

[0154] The knowledge classification module 603 is further used to determine the knowledge type corresponding to each question-answer pair based on the target confusion value threshold range corresponding to each question-answer pair.

[0155] Optionally, the device further includes: a deletion module 605;

[0156] The deletion module 605 is used to delete the question-answer pair when the knowledge type indicates that the question-answer pair is outdated or incorrect knowledge.

[0157] Optionally, the knowledge injection module 604 is further configured to determine the target word from the word elements contained in the question-answer pair, and locate the neuron corresponding to the target word from any layer of the neural network of the target model as the neuron to be adjusted;

[0158] The knowledge injection module 604 is also used to determine the target parameters related to the target word in the neuron to be adjusted by using causal intervention analysis;

[0159] The knowledge injection module 604 is further configured to calculate adjustment parameters based on the target parameters using the least squares method, and replace the target parameters with the adjustment parameters to achieve knowledge update of the target model.

[0160] It should be noted that the specific implementation principle and effect of the knowledge update device based on the industrial internet model can be found in the relevant descriptions and effects of the above embodiments, and will not be elaborated further here.

[0161] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, this application provides an electronic device 700, which includes: a receiver 701, a transmitter 702, a processor 703, and a memory 704.

[0162] Receiver 701 is used to receive commands and data;

[0163] Transmitter 702 is used to send commands and data;

[0164] Memory 704 is used to store instructions executed by the computer;

[0165] The processor 703 is used to execute computer execution instructions stored in the memory 704 to implement the various steps of the knowledge update method based on the industrial internet model in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the knowledge update method based on the industrial internet model.

[0166] Alternatively, the memory 704 can be either standalone or integrated with the processor 703.

[0167] When the memory 704 is set up independently, the electronic device also includes a bus for connecting the memory 704 and the processor 703.

[0168] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the knowledge update method based on the Industrial Internet model as described in any of the foregoing embodiments of this application.

[0169] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the knowledge update method based on the Industrial Internet model as described in any of the foregoing embodiments of this application, which is executed by an electronic device.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0171] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0172] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0173] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0174] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0175] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0176] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0177] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0178] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0179] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0180] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0181] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0182] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0183] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A knowledge update method based on an Industrial Internet of Things (IIoT) model, characterized in that, include: In response to adjustment instructions, multiple question-answer pairs are retrieved from multiple knowledge documents; Determine the perplexity value of the target model for each of the aforementioned question-answer pairs; Based on the confusion value corresponding to each question-answer pair, the knowledge type corresponding to each question-answer pair is determined; When the knowledge type indicates that the question-answer pair is external knowledge, the neurons to be adjusted in the target model are determined based on the question-answer pair, and the parameters of the neurons to be adjusted are replaced to achieve knowledge update of the target model.

2. The method according to claim 1, characterized in that, The process of obtaining multiple question-answer pairs from multiple knowledge documents includes: The knowledge document is input into a pre-trained large language model to obtain multiple question-answer pairs.

3. The method according to claim 1, characterized in that, Determining the perplexity value of the target model for each question-answer pair includes: For each question-answer pair, the question-answer pair is split into multiple consecutive word units; Input the lexical units into the target model to predict the lexical units that follow each of the lexical units; Based on the continuity relationship between each lexical unit, the predicted lexical unit corresponding to each lexical unit, and the total number of lexical units, the perplexity value of each question-answer pair is determined.

4. The method according to claim 1, characterized in that, Each of the aforementioned knowledge types corresponds one-to-one with a pre-set range of confusion value thresholds; The step of determining the knowledge type corresponding to each question-answer pair based on the confusion value of each question-answer pair includes: From the multiple perplexity threshold ranges, the target perplexity threshold range corresponding to the perplexity value of each question-answer pair is matched; Based on the target confusion value threshold range corresponding to each question-answer pair, the knowledge type corresponding to each question-answer pair is determined.

5. The method according to claim 1, characterized in that, The method further includes: When the knowledge type indicates that the question-answer pair is outdated or incorrect knowledge, the question-answer pair is deleted.

6. The method according to claim 1, characterized in that, The step of determining the neurons to be adjusted in the target model based on the question-and-answer pair and replacing the parameters of the neurons to be adjusted to achieve knowledge updating of the target model includes: The target word is determined from the word elements contained in the question-answer pair, and the neuron corresponding to the target word is located in any layer of the neural network of the target model as the neuron to be adjusted. Causal intervention analysis was used to determine the target parameters in the neurons to be adjusted that are related to the target word. Based on the target parameters, the adjustment parameters are calculated using the least squares method, and the target parameters are replaced with the adjustment parameters to achieve knowledge update of the target model.

7. A knowledge updating device based on an industrial internet model, characterized in that, include: The knowledge acquisition module is used to retrieve multiple question-answer pairs from multiple knowledge documents in response to adjustment commands; The determination module is used to determine the perplexity value of the target model for each of the question-answer pairs; The knowledge classification module is used to determine the knowledge type corresponding to each question-answer pair based on the confusion value corresponding to each question-answer pair; The knowledge injection module is used to determine the neurons to be adjusted in the target model based on the question-answer pair when the knowledge type indicates that the question-answer pair is out-of-domain knowledge, and to replace the parameters of the neurons to be adjusted in order to realize the knowledge update of the target model.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images