Large model text reverse reasoning enhancement method based on mask mechanism

Abstract

The invention discloses a large model text reverse reasoning enhancement method based on a mask mechanism, and relates to the technical field of text reasoning, and the method comprises the steps: taking a ModernBERT model as a bottom layer architecture to carry out fine tuning training, construct a knowledge extraction large model, carry out fine-grained knowledge extraction, and determine extracted knowledge; constructing a reverse reasoning instruction data set according to the extracted knowledge; a mask mechanism is introduced, and a prompt word project based on reverse reasoning is constructed; according to the reverse reasoning instruction data set and the cue word project, enhancement training is conducted on the knowledge extraction large model, a knowledge extraction enhancement model is obtained, and reverse reasoning after text masking serves as output of the knowledge extraction enhancement model. The technical problem that in the prior art, in knowledge extraction and text reasoning tasks, an effective reverse reasoning verification capability is lacked, so that the accuracy of a reasoning result is low is solved, and the technical effects of improving the reverse reasoning verification capability of a model in text reasoning and improving the accuracy of the reasoning result are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of text reasoning, and particularly relates to a method for enhancing the reverse reasoning of large model texts based on a masking mechanism. Background Art

[0002] With the rapid development of natural language processing technology, text reasoning and knowledge extraction based on large-scale pre-trained language models have been widely applied in multiple fields. Especially in the automated analysis of scientific and technological literature, it has greatly improved work efficiency. However, existing large model reasoning methods still face significant challenges when dealing with complex causal relationships and reasoning tasks in scientific and technological literature. Current reasoning techniques usually rely on one-way causal derivation, that is, unidirectionally deriving from known input conditions to reasoning results. This method lacks effective reverse reasoning verification ability. Therefore, when the model generates reasoning results, it cannot trace back and verify their rationality through the existing reasoning results, resulting in low accuracy of the reasoning results. Summary of the Invention

[0003] This application provides a method for enhancing the reverse reasoning of large model texts based on a masking mechanism, which is used to solve the technical problem that the existing technology lacks effective reverse reasoning verification ability in knowledge extraction and text reasoning tasks, resulting in low accuracy of reasoning results.

[0004] In view of the above problems, this application provides a method for enhancing the reverse reasoning of large model texts based on a masking mechanism.

[0005] This application provides a method for enhancing the reverse reasoning of large model texts based on a masking mechanism, and the method includes:

[0006] Fine-tuning and training with the ModernBERT model as the underlying architecture to construct a large model for knowledge extraction, performing fine-grained knowledge extraction on scientific and technological literature to determine the extracted knowledge; constructing a reverse reasoning instruction dataset according to the extracted knowledge, where the reverse reasoning instruction dataset is formed in the form of triples; introducing a masking mechanism to construct a prompt engineering based on reverse reasoning, where the masking mechanism is to mask the cause and retain the result, and the prompt engineering is the mapping of the masked statement - original statement before and after masking; performing enhanced training on the large model for knowledge extraction according to the reverse reasoning instruction dataset and the prompt engineering to obtain a knowledge extraction enhanced model, where the knowledge extraction enhanced model takes the reverse reasoning after text masking as the output.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] This application fine-tunes and trains with the ModernBERT model as the underlying architecture to construct a large knowledge extraction model for fine-grained knowledge extraction from scientific and technological literature to determine the extracted knowledge. According to the extracted knowledge, a reverse reasoning instruction dataset is constructed, where the reverse reasoning instruction dataset is formed in the form of triples. A masking mechanism is introduced to construct a prompt engineering based on reverse reasoning, where the masking mechanism is to mask the reason and retain the result, and the prompt engineering is the mapping of the masked statement - original statement before and after masking. According to the reverse reasoning instruction dataset and the prompt engineering, the large knowledge extraction model is enhanced trained to obtain a knowledge extraction enhanced model, where the knowledge extraction enhanced model outputs the reverse reasoning after text masking. The present invention solves the technical problem that the prior art lacks effective reverse reasoning verification ability in knowledge extraction and text reasoning tasks, resulting in low accuracy of reasoning results. By introducing the masking mechanism and the construction and enhanced training of the reverse reasoning instruction dataset, the technical effect of improving the reverse reasoning verification ability of the model in text reasoning and enhancing the accuracy of reasoning results is achieved. Brief Description of the Drawings

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0010] Figure 1 It is a schematic flowchart of a method for enhancing text reverse reasoning of a large model based on a masking mechanism provided by an embodiment of this application;

[0011] Figure 2 It is a schematic flowchart of constructing a large knowledge extraction model in a method for enhancing text reverse reasoning of a large model based on a masking mechanism provided by an embodiment of this application. Detailed Embodiments

[0012] This application provides a method for enhancing text reverse reasoning of a large model based on a masking mechanism to solve the technical problem that the prior art lacks effective reverse reasoning verification ability in knowledge extraction and text reasoning tasks, resulting in low accuracy of reasoning results. By introducing the masking mechanism and the construction and enhanced training of the reverse reasoning instruction dataset, the technical effect of improving the reverse reasoning verification ability of the model in text reasoning and enhancing the accuracy of reasoning results is achieved.

[0013] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0014] It should be noted that any variations of the terms "including" and "having" are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0015] Embodiment, such as Figure 1 As shown, the present application provides a method for enhancing the reverse inference of large model text based on a masking mechanism. The method includes:

[0016] Step S100: Fine-tune and train with the ModernBERT model as the underlying architecture to construct a knowledge extraction large model, perform fine-grained knowledge extraction on scientific and technological literature, and determine the extracted knowledge.

[0017] In the embodiments of the present application, when fine-tuning and training with the ModernBERT model as the underlying architecture to construct a knowledge extraction large model, first obtain artificially labeled samples containing various corpus types, fine-tune and train the ModernBERT model through these samples to generate a classification model for distinguishing different corpus types in scientific and technological literature. Then, based on this classification model, perform semi-automatic classification and annotation on scientific and technological literature, and through manual verification and iterative optimization, determine high-quality fine-tuning corpus. Finally, further train the classification model with the high-quality fine-tuning corpus to construct an accurate knowledge extraction large model, which can effectively identify and extract different types of knowledge.

[0018] Next, perform fine-grained knowledge extraction on scientific and technological literature. Starting from the constructed knowledge extraction large model, automatically identify and extract fine-grained knowledge such as research background, research purpose, research methods, and research results from the target literature. Through this process, determine the extracted knowledge.

[0019] Furthermore, as Figure 2 shown, in the method provided by the embodiments of the application, fine-tuning and training with the ModernBERT model as the underlying architecture to construct a knowledge extraction large model further includes:

[0020] Obtain a corpus identification sample, where the corpus identification sample is an artificially labeled sample and includes multiple corpus types; based on the corpus identification sample, fine-tune and train the ModernBERT model to determine a classification model, where the classification model is used for classifying and identifying corpus types; according to the classification model, perform semi-automatic classification and annotation of scientific and technological literature, verify and iterate on the annotation results to determine high-quality fine-tuning corpus; according to the high-quality fine-tuning corpus, fine-tune and train the classification model to determine a knowledge extraction large model.

[0021] In the embodiment of the present application, first obtain a corpus identification sample from a preset database. These identification samples are manually labeled and include multiple corpus types, such as research background, research purpose, research method, research results, etc.

[0022] Next, based on these corpus identification samples, fine-tune and train the ModernBERT model. ModernBERT is a pre-trained language model based on the Transformer architecture and has powerful language understanding capabilities. During the fine-tuning process, the model adjusts its internal parameters to learn how to classify and identify the corpus types of the literature content according to the input samples. Specifically, the supervised learning method is used for fine-tuning. By inputting the labeled samples, the parameters are adjusted according to the existing labels to learn how to distinguish different types of texts. During the fine-tuning process, the backpropagation algorithm is used to optimize the model parameters so that it can identify and classify different parts in the literature such as research background and research method. Based on the training data, the model can automatically identify the hidden semantic information in the text and match it with the corresponding categories (such as research purpose, research results, etc.). The goal of the fine-tuning training is to enable the model to automatically and accurately classify and extract relevant knowledge points when processing new literature. Through this step, a fine-tuned classification model is finally obtained, where the classification model is used for classifying and identifying corpus types.

[0023] Next, perform automatic annotation of the literature through the classification model. During this process, the classification model automatically classifies the unlabeled scientific and technological literature according to the learned knowledge. Specifically, by inputting the input literature into the classification model, the model performs classification prediction on each paragraph or sentence in the literature and labels it as categories such as "research background" and "research method". However, due to possible errors in automatic annotation, manual verification and correction of the annotation results are carried out, that is, the process of semi-automatic classification annotation. The manual verification personnel verify the results automatically annotated by the model to ensure its accuracy and correct the inaccurate annotations. This process is carried out through iterative optimization to continuously improve the annotation accuracy and finally obtain high-quality annotation data. Through this process, high-quality fine-tuning corpus is finally obtained. These data have undergone multiple rounds of manual verification and optimization to ensure the accuracy and consistency of the annotation.

[0024] After obtaining these high-quality fine-tuning corpora that have been verified and corrected, the classification model is trained and fine-tuned again. Using these high-quality labeled data, the model is retrained to further improve its classification accuracy. Specifically, by inputting the corrected corpus into the model for fine-tuning, the model learns how to better extract information from the literature and judge the correct category of each part according to the context. After multiple rounds of training, the accuracy of the model gradually increases, and finally an efficient large model for knowledge extraction is constructed. Through this process, a model that can accurately identify and extract various fine-grained knowledge in scientific and technological literature is finally obtained. This model can effectively extract information such as research background, research purpose, research methods, and research results.

[0025] Step S200: Construct a reverse inference instruction dataset according to the extracted knowledge, where the reverse inference instruction dataset is formed in the form of triples.

[0026] In the embodiment of the present application, when constructing a reverse inference instruction dataset according to the extracted knowledge, a triple form is used for organization. In this process, first, the representation structure of the dataset is set, and the dataset is constructed using triples (instruction - input - output). Specifically, the instruction part defines the requirements of the inference task and the key basis of the task original text. The input part contains the fine-grained information extracted from the already extracted knowledge (such as research background, research methods, etc.), and the output part is the original text extraction statement obtained through inference. Then, according to this representation structure, the knowledge structure of the extracted knowledge is adjusted to ensure that the relationship between each knowledge point and the reverse inference task is clear. Finally, a reverse inference instruction dataset presented in the form of triples is constructed, and this dataset corresponds to multiple corpus types, covering at least research method types, research purpose types, innovation point types, and research value types, etc.

[0027] Furthermore, in the method provided by the embodiment of the application, constructing a reverse inference instruction dataset according to the extracted knowledge further includes:

[0028] Set the representation structure of the instruction adjustment dataset, where the triple structure of instruction - input - output is used as the representation structure; according to the representation structure, adjust the knowledge structure of the extracted knowledge to construct the reverse inference instruction dataset, where the reverse inference instruction dataset corresponds to multiple corpus types and at least includes research method types, research purpose types, innovation point types, and research value types.

[0029] Furthermore, the method provided by the embodiment of the application further includes:

[0030] The instruction contains the definition of the reasoning task and the key basis of the task original text; the input contains fine-grained knowledge; the output contains the extracted statements from the original text.

[0031] In the embodiments of the present application, when constructing the reverse reasoning instruction dataset according to the extracted knowledge, the representation structure of the instruction adjustment dataset is first set. To clearly express the requirements, input, and output of the reasoning task, a triple structure (instruction - input - output) is used to organize the data. This structure can effectively express all aspects of the reasoning task in an orderly manner, enabling each item in the dataset to clearly reflect the reasoning task, input, and output. Specifically, the instruction part defines the requirements of the reasoning task and the key basis of the task original text. For example, the instruction can be "Infer the purpose of the research based on the following research background information". In this part, the definition of the task clarifies the direction of reasoning and the goal to be achieved. By setting the instruction, it can be ensured that the model knows what kind of reasoning work it needs to perform and the core basis of the reasoning task. The input part contains fine-grained knowledge extracted from the already extracted knowledge, and these knowledge points may include content such as research background, research purpose, research methods, research results, etc. The fine-grained knowledge provides specific context for the reasoning task, helping the model understand the information required in the reasoning process. For example, the "research background" extracted from the literature will provide a basis for inferring the "research purpose". The output part is the extracted statement from the original text derived through the reasoning task. This part reflects the final result of the reasoning task, usually a specific statement extracted from the literature. For example, the output may be "The purpose of this research is to improve system efficiency". Through this step, a clearly defined representation structure, namely the triple structure, including clear reasoning task instructions, input knowledge, and expected output, is obtained, laying a foundation for constructing the dataset.

[0032] Next, according to the representation structure, the knowledge structure of the extracted knowledge is adjusted to ensure a clear correspondence between each knowledge point and the reasoning task. Specifically, the method of manual annotation is used to match different types of knowledge points (such as research method type, research purpose type, innovation point type, research value type, etc.) with the reasoning task. Through annotation, these fine-grained knowledge points are organized into a clear structure to ensure that each knowledge point can play a role in the reverse reasoning task. For example, the research background can correspond to the research purpose reasoning task, and the research method can be associated with the innovation point. Through this structural adjustment, it is ensured that the data can accurately serve the reasoning task and each knowledge point can be effectively utilized. The purpose of the knowledge structure adjustment is to make the connection between each knowledge point and the reasoning task closer, ensuring that the dataset can provide accurate and relevant information during reverse reasoning.

[0033] After completing the knowledge structure adjustment, a reverse reasoning instruction dataset that conforms to the triple structure is finally constructed. This dataset contains multiple triples, each of which contains an instruction, an input, and an output. These triples form the data basis for the reverse reasoning task. Among them, the instruction contains the definition of the reasoning task and the key basis of the task original text, clarifying the task goal; the input contains fine-grained knowledge, which provides necessary context information for the reasoning task; the output part is the result of the reasoning task, which is the original text extraction statement derived from the input. In this way, the constructed reverse reasoning instruction dataset not only has a clear structure but also can support diverse reasoning tasks, ensuring the accuracy and comprehensiveness of the data. These triple data items cover various corpus types, including research method types, research purpose types, innovation point types, and research value types, etc.

[0034] Step S300: Introduce a masking mechanism to construct a prompt engineering based on reverse reasoning, where the masking mechanism is to mask the reason and retain the result, and the prompt engineering is the mapping of the masked statement - the original statement before and after masking.

[0035] In the embodiment of the present application, when introducing a masking mechanism to construct a prompt engineering based on reverse reasoning, first, a special marker is introduced, which is used to mask and mark the reason part in the statement, while the result part remains unchanged. The masking mechanism realizes reason masking and result retention in this way, that is, the reason part in the statement (such as "in order to improve efficiency") is replaced with a special marker (such as "[MASK]"), while the result part (such as "adopted a new technology") remains unchanged. Next, based on this special marker, the original statement in the scientific and technological literature is masked to obtain a masked statement, that is, the incomplete text after masking the reason part. Then, the mapping between the masked statement and the original statement is established, that is, by comparing the masked statement and the original statement, the corresponding relationship between the masked part and the original statement is determined, and finally the prompt engineering is formed.

[0036] Furthermore, in the method provided by the embodiment of the application, introducing a masking mechanism to construct a prompt engineering based on reverse reasoning further includes:

[0037] Introduce a special marker, where the special marker is used to mask and mark the reason part of the statement; according to the special marker, perform statement masking processing on the scientific and technological literature to determine a masked statement, where the masked statement is incomplete text; establish the mapping between the masked statement and the original statement to determine the prompt engineering.

[0038] In the embodiments of the present application, a special marker is first introduced. The purpose is to mask and mark the cause part in the statement to ensure that only the result part is focused on during the reasoning process. By using a marker (such as [MASK]), the cause part in the text (for example, "in order to improve efficiency") is replaced, while the result part (for example, "the research adopted a new technology") remains unchanged. In this way, the cause part is masked, and the reasoning task still focuses on the complete causal relationship. Then, after introducing the special marker, sentence masking processing is performed on the scientific and technological literature, that is, the cause part in the literature is replaced with [MASK], thereby generating a masked sentence. For example, the original sentence "In order to improve efficiency, the research adopted a new technology" becomes "In order to [MASK], the research adopted a new technology" after masking processing. Through this masking processing, the model can focus on the causal reasoning task and concentrate on reasoning the causal relationship of the masked part.

[0039] Next, by establishing a mapping between the masked sentence and the original sentence, the relationship between the masked sentence and the original sentence is made clear. Specifically, through an alignment method, the masked sentence is compared with the original text to ensure that each [MASK] marker corresponds to the cause part in the original sentence. The establishment of this mapping relationship enables the masked part to correctly correspond to the original text, thereby helping the reasoning process to accurately restore the masked content. Through these steps, a prompting engineering based on reverse reasoning is constructed. Among them, the mapping relationship between the masked sentence and the original sentence ensures the smooth progress of the reasoning task, enabling an effective connection to be established between the causal relationship reasoning of the masked part and the known result, thus promoting the completion of the reasoning task.

[0040] Step S400: According to the reverse reasoning instruction dataset and the prompting engineering, perform enhanced training on the knowledge extraction large model to obtain a knowledge extraction enhanced model, where the knowledge extraction enhanced model outputs the reverse reasoning after text masking.

[0041] In the embodiments of the present application, when enhancing the training of the knowledge extraction large model according to the reverse inference instruction dataset and prompt engineering, first, the knowledge extraction large model is enhanced through learning using the inference instruction dataset to obtain a first-order training model. In this stage, the parameters of the model are adjusted so that it can understand and execute inference tasks and learn how to extract effective information from the given input. Next, based on prompt engineering, masked inference and comparison iterative training are performed on the first-order training model. In this process, the masking mechanism masks the cause part in the input, enabling the model to deduce the masked information from the result part. Through multiple rounds of iterative training, the inference ability of the model is enhanced, and finally, a knowledge extraction enhanced model is generated. The enhanced model takes the reverse inference after text masking as the output and can deduce accurate knowledge content according to the masked statement of the input. Especially in the reverse inference task, it ensures the restoration of the masked causal relationship from the masked text.

[0042] Further, in the method provided by the application embodiments, when enhancing the training of the knowledge extraction large model according to the reverse inference instruction dataset and the prompt engineering to obtain a knowledge extraction enhanced model, it further includes:

[0043] Enhancing the learning of the knowledge extraction large model according to the inference instruction dataset to determine a first-order training model; performing masked inference and comparison iterative training on the first-order training model according to the prompt engineering to generate the knowledge extraction enhanced model.

[0044] In the embodiments of the present application, first, the knowledge extraction large model is enhanced through learning using the inference instruction dataset. The inference instruction dataset contains multiple triple structures, where instruction defines the requirements of the inference task and the key basis of the task original text, input is the fine-grained knowledge extracted from the literature, and output is the inference result. To perform enhanced learning, the data is input into the knowledge extraction large model, which is based on the Transformer architecture and applicable to natural language processing tasks. The knowledge extraction large model learns how to generate corresponding inference results according to the input knowledge through a supervised learning method in this stage. The backpropagation algorithm is used to optimize the parameters of the model, enabling the model to identify the relationship between the input and output during the training process and gradually improve the accuracy of inference. Finally, a first-order training model is generated.

[0045] After obtaining the first-order training model, the model is further trained based on prompt engineering. The prompt engineering adopts a masking mechanism, where the cause part in the original text is masked and replaced with the [MASK] symbol, while the result part remains unchanged. In this step, the masked statement is used as the input, and through the inference process of the model, an inference result is generated. Specifically, the first-order training model first receives the masked statement and infers the missing causal relationship part based on the context information. This inference process depends on the parameters obtained in the reinforcement learning stage. The model captures the key information in the input through the self-attention mechanism and infers the masked content.

[0046] Next, by comparing the masked statement with the original statement, the semantic similarity is calculated to ensure the content consistency between the inference result and the original text. The semantic similarity is usually measured by calculating the cosine similarity, which represents the similarity between the inference result and the original statement. If the semantic similarity is low, multi-round iterative training is entered. The model adjusts its parameters to reduce the gap between the inference result and the original statement. During each round of training, the model corrects the inference result of the masked statement according to the calculated similarity feedback, gradually improving the accuracy and precision of the inference until the semantic similarity reaches the set standard.

[0047] Through the above masked inference and comparison iterative training, a knowledge extraction enhanced model is finally generated. This enhanced model can perform backward inference from the masked text to restore the hidden causal relationship. The knowledge extraction enhanced model can deduce the cause part of the causal relationship based on the result part and generate relevant knowledge extraction results. This enhanced training enables the model to more accurately extract fine-grained knowledge, such as research purposes, research methods, innovation points, etc., helping to extract and infer important knowledge from scientific and technological literature.

[0048] Furthermore, in the method provided by the application embodiment, the masked inference and comparison iterative training of the first-order training model further includes:

[0049] Using the masked statement as the input, performing inference based on the first-order training model to determine the inference result; proofreading the masked statement and the original statement to determine the semantic similarity, where the semantic similarity is used as the proofreading standard; and performing multi-round iterative training until convergence according to the semantic similarity to obtain the knowledge extraction enhanced model.

[0050] In the embodiment of the present application, first, a masked statement is used as the input and fed into a first-order training model. A masked statement refers to a situation where the cause part in the original text is replaced with the [MASK] symbol while the result part remains unchanged. By inputting the masked statement, the first-order training model makes inferences based on the context, predicts and generates the masked part. The inference process mainly relies on the parameters obtained by the model during the reinforcement learning stage. At this time, the first-order training model uses its internal self-attention mechanism to deeply analyze the input, thereby inferring the masked causal part and generating an inference result, which is to fill in the content of the [MASK] part.

[0051] Next, the generated inference result is proofread. Specifically, the masked statement is compared with the original statement, and the semantic similarity between the inference result and the original statement is calculated. Semantic similarity is a criterion for measuring the consistency between the inference result and the original text content, and cosine similarity is used for calculation. If the semantic consistency between the inference result and the original statement is high, the semantic similarity is high; conversely, if the semantic similarity is low, it indicates that there is a large difference between the inference result and the original statement, and the model needs to be further optimized. In this stage, semantic similarity serves as the proofreading criterion to help determine whether the inference result of the model meets the accuracy requirements.

[0052] According to the calculated semantic similarity, a multi-round iterative training process is entered. If the semantic similarity between the inference result and the original statement has met the preset convergence criterion, this round of inference task is completed, and the final inference result is output. If the semantic similarity does not reach the expected standard, the model will perform the step of correcting the inference result. The process of correcting the inference result is based on the comparison between the original statement and the inference result. By adjusting the content of the inference result, it is made closer to the semantics of the original text. The corrected inference result will be used as a new input and fed into the model for the next round of training.

[0053] Through continuous iterative training, the inference result of the model is gradually optimized until the semantic similarity meets the set standard. In each round of iteration, the first-order training model adjusts its parameters according to the corrected inference result to optimize the inference process. Through this multi-round iterative training, the model gradually improves the inference accuracy and finally obtains a convergent inference process.

[0054] Finally, after multi-round iterative training, a knowledge extraction enhancement model is generated. This model can perform reverse inference based on the masked statement, restore the masked causal relationship, and extract fine-grained knowledge in scientific and technological literature, such as research purposes, research methods, innovation points, etc. Through reinforcement learning and iterative training, the inference accuracy and accuracy of the model have been significantly improved, and it can better support the knowledge extraction task.

[0055] In summary, by inputting the masked statement into the first-order training model for inference, proofreading with the original statement by calculating the semantic similarity, and finally using multi-round iterative training to correct the inference result until the semantic similarity converges, a knowledge extraction enhancement model is generated.

[0056] Further, in the method provided by the application embodiment, performing multi-round iterative training until convergence according to the semantic similarity further includes:

[0057] Identifying the semantic similarity, and determining whether it meets a preset convergence degree; if it meets, using the inference result as the output; if it does not meet, correcting the inference result based on the original statement to generate a corrected inference result; and performing iterative training on the first-order training model according to the corrected inference result until the semantic similarity is met.

[0058] In the embodiment of the present application, first, the semantic similarity is identified, that is, the semantic similarity between the inference result and the original statement is calculated. The cosine similarity is used to compare the similarity between the inference result and the original statement in the vector space to determine their consistency. If the calculated semantic similarity is greater than the preset convergence threshold, it is considered that the inference result is accurate enough to meet the preset convergence degree, and the inference result is directly used as the output. If the semantic similarity is lower than the preset convergence degree, that is, lower than the preset convergence threshold, it indicates that there is a large gap between the inference result and the original text, and the model needs to be further optimized.

[0059] At this time, it enters the stage of correcting the inference result. First, the inference result and the original statement are compared and analyzed to identify the parts of the inference result that are inconsistent with the original statement. The difference analysis method is used to locate the semantic differences and correct them in combination with the content in the original statement. The correction process modifies the inconsistent content in the inference result to make it closer to the semantics and structure of the original text. For example, if the causal relationship is missing or the logic is unclear in the inference result, the inference result is supplemented or adjusted according to the original statement to ensure more accurate inference. During the correction process, grammar correction techniques are used to adjust the inference result to ensure the correctness and fluency of the language structure.

[0060] After that, the corrected inference result is used as the new input and sent into the first-order training model for iterative training. After each training, its internal parameters are updated through the backpropagation algorithm and the gradient descent method to optimize the inference result. In each round of iteration, the semantic similarity is calculated according to the corrected inference result and compared with the original statement. If the semantic similarity reaches the preset convergence degree, the inference process ends and the final result is output; if the similarity still does not reach the standard, the inference result is continuously corrected and enters the next round of training. Through multiple rounds of iteration, the model gradually improves the inference accuracy and accuracy until the semantic similarity meets the preset convergence degree.

[0061] Finally, through multiple rounds of iterative training and correction, a knowledge extraction enhancement model is generated.

[0062] In the embodiments of the present application, in summary, the embodiments of the present application at least have the following technical effects:

[0063] In the present application, fine-tuning training is carried out with the ModernBERT model as the underlying architecture to construct a large knowledge extraction model for fine-grained knowledge extraction from scientific and technological literature to determine the extracted knowledge; according to the extracted knowledge, a reverse reasoning instruction dataset is constructed, where the reverse reasoning instruction dataset is formed in the form of triples; a masking mechanism is introduced to construct a prompt engineering based on reverse reasoning, where the masking mechanism is to mask the reason and retain the result, and the prompt engineering is the mapping of the masked statement - the original statement before and after masking; according to the reverse reasoning instruction dataset and the prompt engineering, the large knowledge extraction model is enhanced trained to obtain a knowledge extraction enhancement model, where the knowledge extraction enhancement model takes the reverse reasoning after text masking as the output. The present invention solves the technical problem that the prior art lacks effective reverse reasoning verification ability in knowledge extraction and text reasoning tasks, resulting in low accuracy of reasoning results. By introducing the masking mechanism and the construction and enhanced training of the reverse reasoning instruction dataset, the technical effect of improving the reverse reasoning verification ability of the model in text reasoning and improving the accuracy of reasoning results is achieved.

[0064] It should be noted that the above sequence of the embodiments of the present application is only for description and does not represent the superiority or inferiority of the embodiments. And the above specific embodiments of this specification have been described. The processes depicted in the drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0066] This specification and the drawings are only exemplary descriptions of the present application and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalent technologies, the present application is intended to include these changes and modifications.

Claims

1. A large model text reverse reasoning enhancement method based on mask mechanism, characterized in that: The method comprises: Fine-tune the ModernBERT model as the underlying architecture, build a large knowledge extraction model, perform fine-grained knowledge extraction on scientific literature, and determine the extracted knowledge; Constructing a reverse reasoning instruction data set according to the extracted knowledge, wherein the reverse reasoning instruction data set is composed of triples; Introduce a masking mechanism and construct a prompt word engineering based on reverse reasoning, wherein the masking mechanism is cause masking and result retention, and the prompt word engineering is a mapping of masked sentences before and after masking to original sentences; According to the reverse reasoning instruction data set and the prompt word project, the knowledge extraction large model is enhanced trained to obtain a knowledge extraction enhanced model, wherein the knowledge extraction enhanced model uses reverse reasoning after text masking as output.

2. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 1, characterized in that: Fine-tune the ModernBERT model as the underlying architecture and build a large knowledge extraction model, including: Acquire a corpus identification sample, wherein the corpus identification sample is a manually identified sample and includes multiple corpus types; Based on the corpus identification sample, the ModernBERT model is trained and fine-tuned to determine a classification model, wherein the classification model is used to classify and identify the corpus type; According to the classification model, semi-automatic classification annotation of scientific and technological literature is used to verify and iterate the annotation results to determine high-quality fine-tuning corpus; The classification model is trained and fine-tuned based on the high-quality fine-tuning corpus to determine a large knowledge extraction model.

3. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 1, characterized in that: According to the extracted knowledge, a reverse reasoning instruction data set is constructed, including: Setting instructions to adjust the representation structure of the data set, wherein the triple structure of instruction-input-output is used as the representation structure; According to the representation structure, the knowledge structure of the extracted knowledge is adjusted to construct the reverse reasoning instruction data set, wherein the reverse reasoning instruction data set corresponds to a variety of corpus types, including at least research method type, research purpose type, innovation point type and research value type.

4. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 3, characterized in that: The instruction contains the reasoning task definition and the key basis of the task original text; the input contains fine-grained knowledge; and the output contains the original text extraction sentence.

5. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 1, characterized in that: Introduce the mask mechanism and build a prompt word engineering based on reverse reasoning, including: Introducing a special marker, wherein the special marker is used to partially mask the reason of the sentence; According to the special marker, sentence masking is performed on the scientific and technological document to determine a masked sentence, wherein the masked sentence is an incomplete text; A mapping between the masked sentence and the original sentence is established, and the prompt word project is determined.

6. A large model text reverse reasoning enhancement method based on a mask mechanism as claimed in claim 5, wherein the characteristic value is: According to the reverse reasoning instruction data set and the prompt word project, the knowledge extraction large model is enhanced and trained to obtain a knowledge extraction enhanced model, including: According to the inference instruction data set, the knowledge extraction large model is subjected to reinforcement learning to determine a first-order training model; According to the prompt word engineering, mask reasoning and comparison iterative training are performed on the first-order training model to generate the knowledge extraction enhancement model.

7. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 6, characterized in that: The first-order training model is subjected to mask reasoning and comparison iterative training, including: Taking the masked sentence as input, performing reasoning based on the first-order training model, and determining a reasoning result; Proofreading the masked sentence and the original sentence to determine semantic similarity, wherein the semantic similarity is used as a proofreading standard; According to the semantic similarity, multiple rounds of iterative training are performed until convergence to obtain the knowledge extraction enhancement model.

8. A large model text reverse reasoning enhancement method based on mask mechanism as claimed in claim 7, characterized in that: According to the semantic similarity, multiple rounds of iterative training are performed until convergence, including: Identify the semantic similarity and determine whether it satisfies a preset convergence degree; If satisfied, the reasoning result is used as output; If not satisfied, the reasoning result is corrected based on the original sentence to generate a corrected reasoning result; According to the modified reasoning result, the first-order training model is iteratively trained until the semantic similarity is satisfied.

Images