Emergency distortion information identification method and system

Abstract

The invention belongs to the technical field of information analysis, and particularly discloses an emergency distortion information identification method and system. Comprising the following steps: constructing a keyword table of the emergencies according to keywords of the emergencies; screening the emergency distortion information data set so as to construct an emergency distortion information training data set and a test data set; building a first-stage check large model through a prompt strategy of sample embedding, and building a second-stage review large model through a prompt strategy of chain thinking; an XLNet-BiLSTM hybrid model is constructed to identify the distortion information of the emergency event; and constructing a model dynamic integration rule considering the illusion relieving consequence and the specific distortion score, and thus obtaining a distortion information identification result of the two-stage large model and the XLNet-BiLSTM hybrid model. According to the method, the consistency and the understandability of the identification result and the generation explanation are ensured, and the mobility of information identification in different types of emergencies is highlighted.

Description

Technical Field

[0001] The present invention belongs to the field of information analysis technology, and specifically relates to a method and system for identifying distorted information of sudden events, and more specifically, to a method and system for identifying distorted information of sudden events that integrates a two-stage large model with XLNet-BiLSTM. Background Art

[0002] Existing methods for identifying distorted information fall into three categories: statistical methods, machine learning, and deep learning. These methods primarily design algorithms from the perspectives of model evolution, graph propagation, and information features. However, unlike general distorted information, emergency distorted information generally exhibits complex characteristics such as misleadingness, diversity, immediacy, and dynamism. Emergencies are often associated with specific times, locations, and situations, and continuously evolve as events unfold. This imbues the associated distorted information with remarkable immediacy and dynamics. However, due to their reliance on fixed training datasets, traditional identification methods based on information features often face challenges such as limited data coverage and lagging knowledge updates, making them difficult to rapidly adapt to changes in distorted information about emergencies. This has led to limited research on distorted information identification in the emergency context, both domestically and internationally.

[0003] Recent developments in large language model technology have provided new solutions for identifying distorted information. Research has already been conducted on verifying news headlines and rumors using large language models, demonstrating the effectiveness of large models' unique knowledge base, logical reasoning, and semantic understanding capabilities in identifying distorted information. However, most research has overlooked the "hallucination" drawback of large models, where model-generated content is inconsistent with objective facts or user input. This manifests itself in distorted information identification as misjudgments of information and the generation of misleading interpretations. To overcome hallucinations, the most fundamental measure is to verify the training data and align the data of large models to ensure complete data reliability. However, due to the high labor cost, scholars prefer to seek methods to mitigate hallucinations. Ji Z et al. proposed that token probabilities can be used to reflect the uncertainty in the generated text characters, words, and sentences to assess hallucinations. However, this method is generally applicable only to locally deployed large models and cannot be used in a wide range of applications that call APIs. Sun X et al. proposed a self-verification strategy that uses the large model itself as a "hallucination" identifier, but it is impossible to evaluate whether the identifier itself is hallucinating. Manakul P et al. repeatedly sampled text generated by the large model to test the consistency of the test text with the generated text under multiple indicators. However, randomly sampling large amounts of text means a higher computational burden and time cost. Therefore, the following problems remain unresolved: 1) Machine learning algorithms and pre-trained models are limited by the size and type of datasets; 2) Large language models are also distrusted in some studies due to the presence of hallucinations. Summary of the Invention

[0004] To address the above-mentioned deficiencies or improvement needs of the existing technology, the present invention provides a framework for identifying distorted information about emergencies that integrates a two-stage large model with XLNet-BiLSTM. The framework includes: organizing keywords for four types of emergencies to obtain an emergency keyword list; using the emergency keyword list to screen six datasets of distorted information about emergencies from multiple sources in six different fields, followed by data preprocessing to construct training and test datasets for distorted information about emergencies; building a verification large model using a sample embedding prompt strategy; building an evaluation large model using a chain thinking prompt strategy; building a two-stage large model based on a self-verification strategy, and comparing the judgment reasons of the two-stage large model with experimental results to determine whether the large model produces hallucinations or misjudgments; constructing an XLNet-BiLSTM hybrid model to identify distorted information about emergencies; and constructing dynamic model integration rules that consider mitigating hallucination consequences and specific distortion scores to obtain the final recognition results. The present invention demonstrates good accuracy in identifying distorted information about emergencies, ensuring consistency and comprehensibility between the recognition results and the generated explanations, and highlighting the transferability of information recognition in different types of emergencies.

[0005] To achieve the above object, according to one aspect of the present invention, a method for identifying distorted information of an emergency is proposed, comprising the following steps:

[0006] Step 1: Construct a keyword table of emergencies based on the keywords of emergencies;

[0007] Step 2: The emergency event distorted information dataset is screened using the emergency event keyword table, and then data preprocessing is performed to construct the emergency event distorted information training dataset and test dataset;

[0008] Step 3: Use the sample embedding prompt strategy to build the first-stage verification model, and use the chain thinking prompt strategy to build the second-stage review model. By comparing the judgment reasons of the two-stage model with the experimental results, we can determine whether the model has produced hallucinations or misjudgments.

[0009] Step 4: Build an XLNet-BiLSTM hybrid model to identify distorted information of emergencies;

[0010] Step 5: Construct a dynamic model integration rule that takes into account the mitigation of hallucination consequences and specific distortion scores, and use it to obtain the distortion information recognition results of the two-stage large model and the XLNet-BiLSTM hybrid model.

[0011] As a further preference, in step one, the emergency includes: natural disasters, accidental disasters, public health events, and social security events.

[0012] As a further preferred method, in step 2, distorted information datasets from different fields and sources are screened, and then duplicate texts are removed through a similarity algorithm, and finally a distorted information dataset for emergency events is constructed, which is used as the training dataset for each experimental model;

[0013] In addition, considering the time sequence, event distribution and label balance factors, the emergency event distorted information training dataset is screened through the emergency event keyword table and then organized into a test dataset.

[0014] As a further preference, in step three, the first-stage verification model performs information recognition on four parts: sample embedding, source statement, text information and task instructions, and generates verification results and distortion scores, thereby extracting verification labels.

[0015] As a further preference, in step three, the second-stage review model conducts a comprehensive evaluation through five aspects: source statement, text information, verification results, task instructions and chain thinking, and makes adjustments and corrections based on the verification results of the first-stage verification model, and finally generates review results and review labels.

[0016] As a further preferred embodiment, in step 4, the pre-trained model is combined with a deep learning algorithm to construct an XLNet-BiLSTM hybrid model to enhance the bidirectional contextual semantic understanding and feature extraction capabilities of information text. The XLNet model is used to capture the deep semantic features of the information text, and the BiLSTM model is used to learn the contextual regularities in the semantic features, and ultimately identify distorted information.

[0017] As a further preferred embodiment, in step five, the construction of a dynamic integration rule for a model that takes into account the mitigation of hallucination consequences and a specific distortion score includes: when the results of the two-stage large model are inconsistent, or when the distortion scores of the two-stage large model are in the interval [40, 60), selecting the XLNet-BiLSTM model as a replacement to identify the distortion information of the sudden event, thereby dynamically integrating the experimental results of the two.

[0018] Specifically, first determine whether the check model result (Check_label) and the review model result (Review_label) in the two-stage large model are equal. If so, proceed to the next step. If not, set the final result (Final_label) to the XLNet-BiLSTM model result (PLM_label). Then, determine whether the average distortion score of the check model and the review model is within the range [40, 60]. If so, set the final result to the XLNet-BiLSTM model result; if not, set the final result to the two-stage large model result (LLM_label). This concludes the dynamic model integration process.

[0019] According to another aspect of the present invention, there is also provided a system for identifying distorted information of an emergency, comprising:

[0020] The first main control module is used to construct a keyword table of emergency events based on the keywords of the emergency events;

[0021] The second main control module is used to filter the emergency event distortion information data set through the emergency event keyword table and then perform data preprocessing to construct the emergency event distortion information training data set and test data set;

[0022] The third main control module is used to build the first-stage verification model through the prompt strategy of sample embedding, and to build the second-stage review model through the prompt strategy of chain thinking. By comparing the judgment reasons of the two-stage models with the experimental results, it is determined whether the models have produced hallucinations or misjudgments.

[0023] The fourth main control module is used to build an XLNet-BiLSTM hybrid model to identify distorted information of sudden events;

[0024] The fifth main control module is used to construct dynamic model integration rules that take into account the mitigation of hallucination consequences and specific distortion scores, and thereby obtain the distortion information recognition results of the two-stage large model and the XLNet-BiLSTM hybrid model.

[0025] As a further preference, the first-stage verification model conducts information recognition in four parts: sample embedding, source statement, text information and task instructions, and generates verification results and distortion scores, thereby extracting verification labels; the second-stage review model conducts a comprehensive evaluation through five links: source statement, text information, verification results, task instructions and chain thinking, and makes adjustments and corrections based on the verification results of the first-stage verification model, and finally generates review results and review labels.

[0026] As a further preference, in the fourth main control module, the pre-trained model is combined with the deep learning algorithm to construct an XLNet-BiLSTM hybrid model to enhance the bidirectional contextual semantic understanding and feature extraction capabilities of information text. The XLNet model is used to capture the deep semantic features of the information text, and the BiLSTM model is used to learn the contextual rules in the semantic features, and ultimately identify distorted information.

[0027] In general, the above technical solutions conceived by the present invention have the following technical advantages compared with the existing technology:

[0028] Faced with the current lack of emergency text datasets, this paper screens and refines multivariate distortion information datasets based on national standards, compiling them to obtain a distortion information dataset for emergencies. Furthermore, it proposes a distortion information recognition framework that integrates a two-stage large language model with the XLNet-BiLSTM model. The advantages of this framework mainly include the following aspects:

[0029] First, the TLXM framework proposed in this paper achieved the best accuracy in comparative experiments with multiple model algorithms such as machine learning, deep learning, pre-trained models, and large language models. Two sets of model ablation experiments were also designed to explore the contribution of each unit in the framework to the recognition results and the degree of decline in recognition performance after removing each unit, thereby demonstrating the necessity of each unit and the effectiveness of the framework structure.

[0030] Second, the TLXM framework proposed in this invention alleviates the consequences of hallucinations, ensures the consistency of framework recognition results, and uses different prompting strategies to reduce the occurrence of model hallucinations in the generated text of large models, with higher factual consistency, thereby providing an explanatory basis for the framework results.

[0031] Third, the feasibility of the proposed framework in identifying real-time distorted information text was verified through the information text of four types of recent emergencies. It was proved that the framework can be migrated to different emergencies to carry out information recognition, highlighting the application potential and practical value of the framework on real-world Internet platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of a distortion information recognition framework integrating a two-stage large model and XLNet-BiLSTM according to an embodiment of the present invention;

[0033] Figure 2 1 is a schematic diagram of a two-stage large language model reasoning process based on different prompting strategies according to an embodiment of the present invention;

[0034] Figure 3 is a structural diagram of the XLNet-BiLSTM hybrid model involved in an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of a decision tree for dynamic integration rules of a model involved in an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the correct and incorrect distribution ratios of the verification LLM, review LLM, and XLNet-BiLSTM in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of the error distribution of the two-stage large model, the verification LLM, the review LLM, and the XLNet-BiLSTM error distribution in an embodiment of the present invention;

[0038] Figure 7 Schematic diagram of consistency indicators of texts generated under different prompting strategies in an embodiment of the present invention;

[0039] Figure 8 It is a schematic diagram of the TLXM framework portability test results in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0041] Example 1

[0042] like Figure 1 As shown, an embodiment of the present invention provides a method for identifying distorted information about emergencies that integrates a two-stage large model and XLNet-BiLSTM, including: organizing keywords for four types of emergencies to obtain an emergency keyword list; screening six datasets of distorted information about emergencies from multiple sources in six different fields using the emergency keyword list, and then performing data preprocessing to construct training and test datasets for distorted information about emergencies; building a verification large model using a sample embedding prompt strategy; building an evaluation large model using a chain thinking prompt strategy; building a two-stage large model based on a self-verification strategy, and comparing the judgment reasons of the two-stage large model with experimental results to determine whether the large model produces hallucinations or misjudgments; constructing an XLNet-BiLSTM hybrid model to identify distorted information about emergencies; and constructing a dynamic model integration rule that considers the mitigation of hallucination consequences and specific distortion scores to obtain the final recognition result. The present invention demonstrates good accuracy in identifying distorted information about emergencies, ensuring consistency and comprehensibility between the recognition results and the generated explanations, and highlighting the transferability of information recognition in different types of emergencies.

[0043] Specifically, the method includes:

[0044] Step 1: Organize the keywords of four types of emergencies and obtain a keyword table for emergencies.

[0045] In one embodiment of the present invention, in step one, a keyword table corresponding to four types of emergencies is obtained by sorting out the national standard for emergencies (GB / T35561-2017). The four types of emergencies are abbreviated as natural disaster events (N), accident disaster events (A), public health events (H) and social security events (S).

[0046] Step 2: Use the emergency keyword table to screen the emergency distorted information datasets from six different fields and multiple sources, and then perform data preprocessing to construct the emergency distorted information training dataset and test dataset.

[0047] In step 2, six distorted information datasets from different fields and multiple sources were screened, and then duplicate text was removed through a similarity algorithm. Finally, a distorted information dataset (MEE) for emergencies was constructed and used as the training dataset for each experimental model. The training dataset spans the time period from 2009 to 2021, with an overall balanced label distribution. The information text basically covers the main areas of the four types of emergencies to ensure comprehensive knowledge about emergencies. A total of 10,575 data items are included. The original dataset and the distorted information training dataset for emergencies are shown in the following table:

[0048]

[0049] Note: T / F indicates true / false of the label.

[0050] In addition, considering the time sequence, event distribution and label balance factors, the emergency event keyword table was used to filter the emergency event distorted information training data set, and then organized into a test data set with a total of 1096 data.

[0051]

[0052] Note: T / F indicates true / false of the label.

[0053] Step 3: Build a large verification model through the prompt strategy of sample embedding. In this step, the first-stage verification large model performs information recognition based on four parts: sample embedding, source statement, text information, and task instructions, and generates verification results and distortion scores, thereby extracting verification labels. More specifically, in this step, the first-stage verification large model adopts a prompt strategy of few-sample embedding, because the prompt strategy of sample embedding can provide a small number of examples to the large model, including task descriptions and expected results, which can improve the initial accuracy of the entire framework and enable the large model to quickly clarify task intent and output standards. The first-stage verification large model performs information recognition based on four parts: sample embedding, source statement, text information, and task instructions, and generates verification results and distortion scores, thereby extracting verification labels.

[0054] Step 4: Build the review model using a chain-thinking prompting strategy. In this step, the second-stage review model comprehensively evaluates the source statement, text information, verification results, task instructions, and chain-thinking. Adjustments and corrections are made based on the verification results of the first-stage verification model, ultimately generating the review results and review labels. More specifically, in this step, the second-stage review model uses a chain-thinking prompting strategy, which is divided into five thinking steps: clarifying emotional tendencies, event types, logical structures, verification explanations, and sentence structure specifications. This chain-thinking prompting strategy allows the model to think step by step when generating responses, breaking down complex tasks into simpler ones. It also allows the task steps to be clearly defined in the prompts, limiting the dimensions of the model's thinking, thereby reducing the probability of the model generating hallucinations and the randomness of the output. The second-stage review model comprehensively evaluates the source statement, text information, verification results, task instructions, and chain-thinking. Adjustments and corrections are made based on the verification results of the first-stage verification model, ultimately generating the review results and review labels.

[0055] Step 5: Build a two-stage large model based on the self-verification strategy, and judge whether the large model produces hallucinations or misjudgments by comparing the judgment reasons of the two-stage large model with the experimental results. In this step, the two-stage large model reasoning process based on different prompting strategies is shown in the figure. Figure 2 As shown above, the distortion score given by the review model is 10 points lower than that of the verification model, and the self-verification results are consistent. However, for other distorted information texts, the labels of the two-stage large model may be inconsistent. Therefore, the present invention believes that the results generated by the large language model may contain hallucinations and misjudgments. Therefore, the accuracy of the recognition results cannot be guaranteed by the large language model alone, and it needs to be considered comprehensively in combination with the pre-trained model.

[0056] Step 6: Construct an XLNet-BiLSTM hybrid model to identify distorted information about emergencies. This model performs judgment migration based on the distortion of the judgment of the two-stage large model mentioned above, and uses the XLNet-BiLSTM hybrid model to identify distorted information about emergencies. In this step, the pre-trained model is combined with the deep learning algorithm to construct an XLNet-BiLSTM hybrid model to enhance the two-way contextual semantic understanding and feature extraction capabilities of the information text. The XLNet model can capture the deep semantic features of the information text, and the BiLSTM model can learn the contextual laws in the semantic features, and finally identify the distorted information. The structure of the XLNet-BiLSTM hybrid model is as follows: Figure 3 In this step, based on the dependence of pre-trained models and deep learning algorithms on training datasets, when large language models have the potential for hallucination, the XLNet-BiLSTM model based on a fixed dataset can be considered as an alternative to improve the reliability and accuracy of the recognition framework.

[0057] Step 7: Construct a dynamic model integration rule that takes into account the mitigation of hallucination consequences and specific distortion scores to obtain the final recognition result. In this step, the model integration rule is constructed. When the results of the two-stage large model are inconsistent, or the distortion score of the large model is in the range of [40, 60), the XLNet-BiLSTM model is selected as a replacement, thereby dynamically integrating the experimental results of the two. The specific process is as follows: Figure 4 shown.

[0058] Specifically, first determine whether the check model result (Check_label) and the review model result (Review_label) in the two-stage large model are equal. If so, proceed to the next step. If not, set the final result (Final_label) to the XLNet-BiLSTM model result (PLM_label). Then, determine whether the average distortion score of the check model and the review model is within the range [40, 60]. If so, set the final result to the XLNet-BiLSTM model result; if not, set the final result to the two-stage large model result (LLM_label). This concludes the dynamic model integration process.

[0059] Based on the above rules, on the one hand, the present invention can timely correct the results of the two-stage large model through the results of the XLNet-BiLSTM model to minimize the influence of hallucinations. On the other hand, from the perspective of specific distortion scores, the present invention considers the advantage intervals of the two-stage large model and the XLNet-BiLSTM model in identifying distortion information and makes a trade-off, thereby enhancing the robustness of distortion information recognition, realizing dynamic model integration and obtaining the final prediction result.

[0060] Example 2

[0061] refer to Figure 1 The present invention proposes a distortion information recognition framework that integrates a two-stage large model with XLNet-BiLSTM. The specific implementation method of the invented method is described in detail below:

[0062] Step 1: Predetermine the model and model parameters to be used.

[0063] Step 2: Given the above conditions, write code, construct the model framework proposed by this method, and identify emergency information.

[0064] In order to simplify the text of this application and reduce its length, the specific schemes and formulas involved here can be found in the invention content section of the previous specification and will not be repeated here.

[0065] Example 1

[0066] The following is specifically described based on Example 1:

[0067] This paper selected GLM-4-Air, released by Zhipu Qingyan, one of the top three domestic large-scale models in a certain period, as its experimental model. It constructed a two-stage large-scale model structure and performed information recognition by calling the API interface. Furthermore, this paper used XLNet's benchmark Chinese model "chinese-xlnet-base" and the BiLSTM model from the Pytorch package to build the XLNet-BiLSTM model on the Google Colab platform. The model parameters are shown in the following table:

[0068]

[0069] Note: Type (type); Parameter (parameter).

[0070] Based on the given parameter conditions above, code is written to construct the model framework proposed in this invention and identify emergency information.

[0071] In order to verify the performance of the model framework of the present invention in distorted information recognition, a comparative experiment was designed. The experimental results are shown in the following table:

[0072]

[0073] Note: The evaluation indicators are Accuracy, Precision, Recall, and F1-score.

[0074] As shown in the table above, the proposed TLXM framework achieved optimal results in the distorted information recognition experiment, with accuracy, precision, recall, and F1 scores reaching 85.02%, 84.97%, 85.41%, and 84.97%, respectively. This demonstrates that the proposed TLXM framework significantly outperformed other models in the experimental results, as it leverages the advantages of both a large language model and a pre-trained model.

[0075] To explore the contribution of each unit in the TLXM framework to the final information recognition accuracy, we designed two types of ablation experiments. The first one shows the information recognition results of each unit in the framework on the test set. The results are shown in the following table:

[0076]

[0077] Note: The evaluation indicators are Accuracy, Precision, Recall, and F1-score.

[0078] As shown in the table above, the XLNet-BiLSTM hybrid model achieves superior information recognition results compared to both the XLNet and BiLSTM models alone, demonstrating the importance of combining the two. Furthermore, the accuracy of the XLNet-BiLSTM, the large verification model, and the large review model all fall short of the TLXM framework. This demonstrates that the proposed method of integrating the two-stage large model structure with the XLNet-BiLSTM model leverages the strengths of each model while mitigating its drawbacks, ultimately improving the accuracy of the final recognition results.

[0079] Second, we compared the information recognition performance of the TLXM framework on the test set after removing each unit. The results are shown in the following table:

[0080]

[0081] Note: The evaluation indicators are Accuracy, Precision, Recall, and F1-score.

[0082] As shown in the table above, the ablation combination excluding the XLNetBiLSTM, verification model, and review model units lags behind the TLXM framework in accuracy by 1.10%, 1.37%, and 3.75%, respectively, demonstrating the indispensability of each unit and the rationale behind the TLXM framework. The ablation combination retaining only the two-stage large model structure achieves the closest results to the TLXM framework, highlighting the unique importance of the large language model within the framework. However, its accuracy is lower than that of the review model alone. This is because the two large models may produce hallucinations, leading to conflicting results. Therefore, using the XLNet-BiLSTM model as a replacement for the two-stage large model structure is reasonable, and experimental results show that this also helps improve the accuracy of the final recognition results.

[0083] In order to verify the rationality and effectiveness of the dynamic integration rules, the MEE dataset was used to verify and analyze the two links of the rules. When the results of the two-stage large model were inconsistent, the correct and incorrect distribution ratios of the large model, the review large model, and the XLNet-BiLSTM model were verified as follows: Figure 5 As shown:

[0084] As shown in the figure, the XLNet-BiLSTM model achieves an accuracy of 73%, significantly higher than the two large language models. This inconsistency is partly due to hallucinations, but also likely stems from the randomness of the large model. Therefore, when the results of the two-stage large models are inconsistent, choosing the XLNet-BiLSTM model that performs consistently with the MEE dataset as an alternative ensures consistent recognition results and effectively mitigates the consequences of hallucinations.

[0085] Figure 6 The following diagram shows the error of the two-stage large model and the XLNet-BiLSTM model at different distortion score ranges:

[0086] As can be seen from the figure, within the distortion score range of 40-60, the error rate of the two-stage large model is significantly higher than in other ranges. In the same score range, the XLNet-BiLSTM model consistently produces fewer errors than the two large models. In other distortion score ranges, the accuracy of the two-stage large model is significantly higher than the error rate, and the XLNet-BiLSTM model also produces significantly more errors than the two-stage large model.

[0087] The present invention constructs a two-stage large model structure by adopting different prompting strategies. However, the combination of prompting strategies is not unique. Therefore, the present invention designs four prompting strategies: "zero-sample embedding", "zero-sample + thinking chain", "few-sample embedding" and "few-sample + thinking chain". The following table shows the recognition effect of the GLM-4-Air model on the MEE dataset under the guidance of different prompting strategies.

[0088]

[0089] Note: The evaluation indicators are Accuracy, Precision, Recall, and F1-score.

[0090] As can be seen from the table above, the three large models using the prompt strategy all achieved certain performance improvements compared to the zero-shot embedding scenario because they provided standard examples, enabling them to more clearly address task requirements and understand the verification perspective more clearly. Among them, the group using the few-shot embedding strategy achieved the best accuracy.

[0091] In order to evaluate the degree of reduction in the probability of hallucination under different prompting strategies, the present invention uses three methods, ROUGE, BERTScore, and FactCC based on natural language reasoning, for analysis. ROUGE focuses on judging the vocabulary overlap of generated texts under different prompting strategies, while BERTScore and FactCC focus on the semantic similarity and factual consistency between the generated text and the original information text. The greater the vocabulary overlap between the generated texts, the higher the semantic similarity with the original text, and the higher the confidence score of the factual consistency, the higher the consistency between the two, and the less likely hallucination will occur. The results are as follows: Figure 7 As shown:

[0092] As can be seen from the figure above, the three metrics of the text generated by the large model based on the chain thinking strategy (cot) are higher than those of the other two groups. Furthermore, in the FactCC test, the text generated by the chain thinking strategy and the original text all have an entailment relationship, while the other two groups contain texts with partial contradictions and neutral relationships. This shows that the chain thinking strategy can effectively improve the similarity of the original text information in the generated text and maintain a high degree of factual consistency with the key content of the original text, thereby reducing the possibility of introducing irrelevant or erroneous information. This is because the chain thinking strategy clarifies the content of each step of the large model's reasoning, gradually considering the degree of distortion of the information text from multiple aspects such as emotional polarity and intensity, event type and theme, causal logic, contextual structure, and data facts. This allows the large model to think step by step based on the original text information, avoiding arbitrary conclusions based on specific content and reducing the possibility of the large model's thinking jumping off point to associate with other information. This contributes to reducing the probability of hallucinations and ensures the consistency and credibility of the text generated by the large model.

[0093] In order to verify the portability of the TLXM framework in the complex environment of the Internet platform after an emergency occurs, this paper selects typical cases for four types of emergencies, including natural disasters, accidents and disasters, and social security. The information texts are crawled on Weibo according to the corresponding keywords and manually labeled. This paper selects the information texts related to emergencies in the WBV1 dataset as the event cases for portability testing. The test results are as follows: Figure 8 shown.

[0094] As can be seen from the above figure, the TLXM framework has an accuracy rate of 96.56%, 95.89%, 94.05% and 94.47% in the four types of emergency information texts, respectively, reflecting the accuracy and transferability of the TLXM framework proposed in this invention in identifying different event information texts.

[0095] Although the present invention has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be considered to provide a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively encompassing the intended scope of the invention. In addition, the above description of the invention is based on the embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the invention that are not currently foreseen may still represent equivalent modifications of the invention.

[0096] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying distorted information of an emergency, characterized in that: The following steps are involved: Step 1: Construct a keyword table of emergencies based on the keywords of emergencies; Step 2: The emergency event distorted information dataset is screened using the emergency event keyword table, and then data preprocessing is performed to construct the emergency event distorted information training dataset and test dataset; Step 3: Use the sample embedding prompt strategy to build the first-stage verification model, and use the chain thinking prompt strategy to build the second-stage review model. By comparing the judgment reasons of the two-stage model with the experimental results, we can determine whether the model has produced hallucinations or misjudgments. Step 4: Build an XLNet-BiLSTM hybrid model to identify distorted information of emergencies; Step 5: Construct a dynamic model integration rule that takes into account the mitigation of hallucination consequences and specific distortion scores, and use it to obtain the distortion information recognition results of the two-stage large model and the XLNet-BiLSTM hybrid model.

2. The method for identifying distorted information of an emergency event according to claim 1, wherein: In step 1, the emergencies include: natural disasters, accidental disasters, public health events, and social security events.

3. The method for identifying distorted information of an emergency event according to any one of claims 1 or 2, characterized in that: In step 2, we screened distorted information datasets from different fields and sources, then removed duplicate texts using a similarity algorithm. Ultimately, we constructed a distorted information dataset tailored to emergencies, which served as the training dataset for each experimental model. In addition, considering the time sequence, event distribution and label balance factors, the emergency event distorted information training dataset is screened through the emergency event keyword table and then organized into a test dataset.

4. A method for identifying distorted information of an emergency event according to any one of claims 1 to 3, characterized in that: In step three, the first-stage verification model performs information recognition in four parts: sample embedding, source statement, text information and task instructions, and generates verification results and distortion scores to extract verification labels.

5. A method for identifying distorted information of an emergency event according to any one of claims 1 to 4, characterized in that: In step three, the second-stage review model conducts a comprehensive evaluation through five aspects: source statement, text information, verification results, task instructions and chain thinking, and makes adjustments and corrections based on the verification results of the first-stage verification model, and finally generates the review results and review labels.

6. A method for identifying distorted information of an emergency event according to any one of claims 1 to 5, characterized in that: In step 4, the pre-trained model is combined with a deep learning algorithm to construct an XLNet-BiLSTM hybrid model to enhance the bidirectional contextual semantic understanding and feature extraction capabilities of information text. The XLNet model is used to capture the deep semantic features of information text, and the BiLSTM model is used to learn the contextual regularities in the semantic features, ultimately identifying distorted information.

7. A method for identifying distorted information of an emergency event according to any one of claims 1 to 6, characterized in that: In step 5, the dynamic integration rules of the model that considers the mitigation of hallucination consequences and specific distortion scores include: when the results of the two-stage large model are inconsistent, or when the distortion scores of the two-stage large model are in the range of [40, 60), the XLNet-BiLSTM model is selected as an alternative to identify the distortion information of the sudden event, thereby dynamically integrating the experimental results of the two.

8. A system for identifying distorted information of emergencies, characterized in that: include: The first main control module is used to construct a keyword table of emergency events based on the keywords of the emergency events; The second main control module is used to filter the emergency event distortion information data set through the emergency event keyword table and then perform data preprocessing to construct the emergency event distortion information training data set and test data set; The third main control module is used to build the first-stage verification model through the sample embedding prompt strategy, and the second-stage review model through the chain thinking prompt strategy. By comparing the judgment reasons of the two-stage model with the experimental results, it is determined whether the model has produced hallucinations or misjudgments. The fourth main control module is used to build an XLNet-BiLSTM hybrid model to identify distorted information of sudden events; The fifth main control module is used to construct dynamic model integration rules that take into account the mitigation of hallucination consequences and specific distortion scores, and thereby obtain the distortion information recognition results of the two-stage large model and the XLNet-BiLSTM hybrid model.

9. The system for identifying distorted information of an emergency event according to claim 8, characterized in that: The first phase of the verification model performs information recognition based on four parts: sample embedding, source statement, text information, and task instructions. It then generates verification results and distortion scores, thereby extracting verification labels. The second-stage review model conducts a comprehensive evaluation through five aspects: source statement, text information, verification results, task instructions and chain thinking, and makes adjustments and corrections based on the verification results of the first-stage verification model, and finally generates review results and review labels.

10. The system for identifying distorted information of an emergency event according to any one of claims 8 or 9, characterized in that: In the fourth main control module, the pre-trained model is combined with a deep learning algorithm to construct an XLNet-BiLSTM hybrid model to enhance the bidirectional contextual semantic understanding and feature extraction capabilities of information text. The XLNet model is used to capture the deep semantic features of information text, and the BiLSTM model is used to learn the contextual patterns in the semantic features, ultimately identifying distorted information.

Images