Text data verification method and related device
By extracting contextual information from text data and using a large model for hierarchical verification, the problem of difficulty in identifying implicit semantic errors in existing technologies is solved, achieving highly accurate text data verification and automated error analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LAUNCH TECH CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-03
AI Technical Summary
Existing text data validation methods based on rule engines struggle to identify implicit semantic errors, resulting in low overall validation accuracy.
By extracting contextual information from text data and combining it with a large model for compliance, consistency, and logical verification, the final verification result is generated. A hierarchical verification strategy is then used to systematically detect both explicit and implicit errors.
It significantly improves the accuracy of text data validation, achieves full coverage of explicit and implicit errors, and provides structured error reports and correction suggestions.
Smart Images
Figure CN122334243A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a text data verification method and related apparatus. Background Technology
[0002] With the development of the digital age, text data has become a core data asset in fields such as enterprise decision-making, government office work, and scientific research and innovation. The requirements for its accuracy and compliance are also increasing, making text data verification a key link in the data processing process.
[0003] Currently, text data validation methods based on rule engines mainly rely on manually preset fixed rules, making it difficult to identify implicit semantic errors in text, such as the implicit logical contradiction in the quantity of "product unit price 20 yuan, total price 100 yuan, purchase quantity 6 pieces". This method can only detect explicit errors that conform to preset rules, resulting in low overall validation accuracy. Summary of the Invention
[0004] This application provides a text data verification method and related apparatus, with the aim of improving the overall verification accuracy.
[0005] To achieve the above objectives, this application provides the following technical solution:
[0006] The first aspect of this application provides a text data verification method, including:
[0007] Extract the context information of each sentence from the original text data, and construct text association data based on the context information of each sentence and the original text data;
[0008] The original text data is validated using the compliance provisions of the domain in which it is located, resulting in a first validation result;
[0009] The original text data is validated using associated reference data to obtain a second validation result;
[0010] Perform text logic verification on the text-related data to obtain a third verification result;
[0011] Based on the first verification result, the second verification result, and the third verification result, the final verification result of the original text data is generated.
[0012] Optionally, the step of verifying the original text data using compliance clauses in the domain in which the original text data resides to obtain a first verification result includes:
[0013] The compliance clauses of the domain in which the original text data is located and the original text data are input into the large model to obtain the semantic similarity score between the original text data and the compliance clauses.
[0014] Determine whether the semantic similarity score is greater than the similarity score threshold;
[0015] If the semantic similarity score is greater than the similarity score threshold, then the original text data is determined to have passed the compliance verification, and a first verification result is generated;
[0016] If the semantic similarity score is not greater than the similarity score threshold, then the original text data is determined to have failed the compliance verification.
[0017] Mark the non-compliant content that fails the compliance check in the original text data and associate the non-compliant content with the corresponding compliance clauses;
[0018] Based on the conclusion that the original text data failed the compliance check, the violation content, and the corresponding compliance clauses, a first verification result is generated.
[0019] Optionally, the step of verifying the original text data using associated reference data to obtain a second verification result includes:
[0020] The original text data is input into the large model to obtain the key entities in the original text data;
[0021] The key entities are compared with the entities in the associated reference data;
[0022] If the key entity is consistent with the entity in the associated reference data, then the original text data is determined to have passed the consistency check, and a second check result is generated.
[0023] If the key entity is inconsistent with the entity in the associated reference data, then the original text data is determined to have failed the consistency check.
[0024] In the original text data, mark the error locations and conflicting key entities that failed the consistency check;
[0025] A second verification result is generated based on the conclusion that the original text data failed the consistency check, the error location, and the conflicting key entity.
[0026] Optionally, the step of performing text logic verification on the text-related data to obtain a third verification result includes:
[0027] The text association data is input into the large model to obtain the logical verification results;
[0028] If the logical verification result indicates that the text association data has a logical contradiction or semantic ambiguity, then it is determined that the text association data has failed the logical verification.
[0029] Mark the locations of logical errors and logically contradictory text in the text association data;
[0030] A third verification result is generated based on the conclusion that the text association data failed the logical verification, the location of the logical error, and the text with logical contradictions.
[0031] If the text logic verification result indicates that the text association data does not have logical contradictions or semantic ambiguities, then the text association data is determined to have passed the logic verification, and a third verification result is generated.
[0032] Optionally, generating the final verification result of the original text data based on the first verification result, the second verification result, and the third verification result includes:
[0033] The error locations and error types that failed the verification are filtered out from the first verification result, the second verification result, and the third verification result; each of the first verification result, the second verification result, and the third verification result corresponds to an error type.
[0034] Input the correct criteria and the error content contained in the error location into the large model to obtain the cause of the error;
[0035] Input the error type and error cause into the large model to obtain correction suggestion information;
[0036] The final verification result of the original text data is generated based on the error location, the error cause, the error type, and the correction suggestion information.
[0037] Optionally, the adjustment process of the large model includes:
[0038] Acquire a pre-trained large model and fine-tuned text data; the fine-tuned text data includes correct text data and incorrect text data; the incorrect text data includes error types; the error types include at least compliance errors, consistency errors, and logical errors;
[0039] The fine-tuned text data is labeled to obtain a fine-tuned dataset;
[0040] Based on the fine-tuning dataset, the pre-trained large model is adjusted using a parameter-efficient fine-tuning method to obtain the adjusted large model.
[0041] Adjust the parameters of the adjusted large model according to the current verification requirements to obtain the large model.
[0042] Optionally, before extracting the context information of each sentence in the original text data, the method further includes:
[0043] The text data to be processed is cleaned to obtain cleaned text data.
[0044] The cleaned text data is then standardized to obtain the original text data.
[0045] A second aspect of this application provides a text data verification device, comprising:
[0046] An extraction unit is used to extract the context information of each sentence in the original text data, and to construct text association data based on the context information of each sentence and the original text data;
[0047] The first verification unit is used to verify the original text data using the compliance clauses of the domain in which the original text data is located, and to obtain a first verification result;
[0048] The second verification unit is used to verify the original text data using associated reference data to obtain a second verification result;
[0049] The third verification unit is used to perform text logic verification on the text-related data and obtain the third verification result.
[0050] The generation unit is used to generate the final verification result of the original text data based on the first verification result, the second verification result, and the third verification result.
[0051] A third aspect of this application provides a computer-readable storage medium comprising a stored program, wherein the program, when executed by a processor, performs the text data verification method provided in the first aspect of this application.
[0052] A fourth aspect of this application provides an electronic device, comprising: a processor, a memory, and a bus; wherein the processor and the memory are connected via the bus;
[0053] The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the text data verification method provided in the first aspect of this application.
[0054] The technical solution provided in this application extracts the contextual information of each sentence in the original text data and constructs text-related data based on the contextual information of each sentence and the original text data. It then verifies the original text data using compliance clauses in the domain in which the original text data resides, obtaining a first verification result. Next, it verifies the original text data using related reference data, obtaining a second verification result. Finally, it performs text logic verification on the text-related data, obtaining a third verification result. Based on the first, second, and third verification results, it generates the final verification result of the original text data. By combining a layered verification strategy, it systematically detects compliance, consistency, and logical errors sequentially, achieving full coverage of both explicit and implicit errors and significantly improving the overall verification accuracy. Attached Figure Description
[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0056] Figure 1 A flowchart of a text data verification method provided in this application embodiment;
[0057] Figure 2 This is a schematic diagram of the architecture of a text data verification device provided in an embodiment of this application;
[0058] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0060] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0061] As Figure 1 shown, it is a flowchart of a text data verification method provided by an embodiment of the present application, including the following steps:
[0062] S101: Extract the context information of each sentence in the original text data, and construct text association data based on the context information of each sentence and the original text data.
[0063] Optionally, a semantic window extraction algorithm can be used to extract the context information of each sentence in the original text data.
[0064] Specifically, the semantic window extraction algorithm aims to dynamically construct a context window containing the context information before and after each sentence in a long text. By extracting N sentences before and after each sentence (N can be dynamically adjusted according to the text length, with a default value of 3), text association data centered on the sentence is formed. This process can automatically equip each sentence with a text segment with a complete semantic atmosphere, thereby providing structured context support for a large-scale language model to understand the detailed semantics, reference relationships, and logical coherence in the text, and enhancing the model's ability to capture the deep semantics of long texts.
[0065] Optionally, before step S101, it is also necessary to preprocess the text data to be processed first, so as to provide high-quality input data for subsequent verification. Therefore, in another embodiment of the present application, a data processing method is provided, including processes A1 to A2.
[0066] A1: Clean the text data to be processed to obtain the cleaned text data.
[0067] Optionally, a character filtering algorithm can be used to remove redundant characters (such as special symbols, whitespace characters, repeated characters) and invalid information (such as advertisement pop-up text, unconcerned annotations) in the text data to be processed, and at the same time, preliminarily correct the spelling mistakes in the text data to be processed based on the matching of a common word library.
[0068] A2: Standardize the cleaned text data to obtain the original text data.
[0069] Specifically, according to the preset text format specifications, the cleaned text data is uniformly processed, including unifying the date format (such as unifying "2024.05.20" and "2024-05-20" to "May 20, 2024"), unifying the measurement units (such as unifying "kg" and "kilogram" to "kilogram"), standardizing the font and font size, etc., to obtain the original text data.
[0070] S102: Verify the original text data using the compliance terms in the field where the original text data is located to obtain the first verification result.
[0071] Understandably, the original text data is validated using the compliance provisions of the domain in which it is located. In other words, the original text data is validated to see if it complies with the compliance requirements of the domain. If it does, the first validation result is that the original text data passes the compliance validation; if it does not, the first validation result is that the original text data fails the compliance validation.
[0072] For example, compliance clauses include laws and regulations, industry standards, and internal corporate rules.
[0073] Optionally, in another embodiment of this application, the specific implementation of step S102 includes processes B1 to B6.
[0074] B1: Input the compliance terms of the domain in which the original text data is located and the original text data into the large model to obtain the semantic similarity score between the original text data and the compliance terms.
[0075] Specifically, the compliance clauses in the domain of the original text data and the original text data are input into the large model. The semantic similarity matching algorithm of the large model is used to calculate the semantic similarity score between the original text data and the compliance clauses.
[0076] B2: Determine whether the semantic similarity score is greater than the similarity score threshold.
[0077] If the semantic similarity score is greater than the similarity score threshold, then process B3 is executed; if the semantic similarity score is not greater than the similarity score threshold, then process B4 is executed.
[0078] The similarity score threshold includes, but is not limited to, 0.8.
[0079] B3: Determine if the original text data passes the compliance check and generate the first check result.
[0080] B4: It has been determined that the original text data failed the compliance check.
[0081] B5: Mark the non-compliant content that fails the compliance check in the original text data and associate it with the corresponding compliance clauses.
[0082] Specifically, when the semantic similarity score is not greater than the similarity score threshold, there is a potential violation. In this case, the violation location and its corresponding compliance clause will be marked in the original text data.
[0083] B6: Generate the first verification result based on the conclusion that the original text data failed the compliance verification, the non-compliant content, and the corresponding compliance clauses.
[0084] S103: Verify the original text data using the associated reference data to obtain the second verification result.
[0085] It is understandable that using related reference data (such as historical text and reference data) to verify the original text data means verifying the consistency between the related reference data and the original text data, including information consistency (such as consistency of key information such as name, ID number, amount, etc.), format consistency, etc.
[0086] Optionally, in another embodiment of this application, the specific implementation of step S103 includes processes C1 to C6.
[0087] C1: Input the original text data into the large model to obtain the key entities in the original text data.
[0088] Specifically, key entities include, but are not limited to, names of people, places, amounts, and dates.
[0089] C2: Compare the key entity with the entity in the associated reference data.
[0090] Specifically, the extracted key entities are semantically compared with the same key entities in other locations within the text and entities in the associated reference data to verify consistency.
[0091] C3: If the key entity is consistent with the entity in the associated reference data, then the original text data is determined to have passed the consistency check, and a second check result is generated.
[0092] Understandably, if the key entity is consistent with the entity in the associated reference data, it means that the key entity matches the associated reference data, and at this time, the original text data is determined to have passed the consistency check.
[0093] C4: If the key entity is inconsistent with the entity in the associated reference data, then the original text data has failed the consistency check.
[0094] Understandably, if the key entity is inconsistent with the entity in the associated reference data, it means that the key entity and the associated reference data do not match, and in this case, it is determined that the original text data has failed the consistency check.
[0095] C5: In the original text data, mark the error locations and conflicting critical entities that failed the consistency check.
[0096] Among them, the conflict critical entity refers to an entity that is inconsistent with the entity in the associated reference data.
[0097] C6: Generate a second verification result based on the conclusion that the original text data failed the consistency check, the error location, and the conflicting key entities.
[0098] S104: Perform text logic validation on the text-related data to obtain the third validation result.
[0099] It is understandable that performing text logic validation on text-related data means checking for hidden errors such as logical contradictions and semantic ambiguities in the original text data.
[0100] Optionally, in another embodiment of this application, the specific implementation of step S104 includes processes D1 to D5.
[0101] D1: Input the text association data into the large model to obtain the logical verification results.
[0102] It is understandable that textual data is input into a large model, and the model's logical reasoning ability is used to analyze the logical relationships in the text (such as causal relationships, parallel relationships, conditional relationships, etc.) to obtain logical verification results.
[0103] D2: If the logical verification result indicates that there is a logical contradiction or semantic ambiguity in the text association data, then it is determined that the text association data has failed the logical verification.
[0104] For example, a logical contradiction is that the quantity purchased is 5 items, the unit price is 10 yuan, and the total price is 60 yuan; semantic ambiguity is that a polysemous word does not have a clear referent.
[0105] D3: Mark logical errors and logically contradictory text in the text association data.
[0106] D4: Generate a third verification result based on the conclusions, logical error locations, and logical contradictions in the text-related data that failed the logical verification.
[0107] D5: If the text logic verification result indicates that there are no logical contradictions or semantic ambiguities in the text association data, then the text association data is determined to have passed the logic verification, and a third verification result is generated.
[0108] As can be seen from the above, by adopting a progressive verification process of "compliance → consistency → logic", surface compliance errors are quickly screened first, and then consistency conflicts and deep logical fallacies are gradually detected. This significantly improves the comprehensiveness and accuracy of the verification while ensuring efficiency.
[0109] S105: Based on the first verification result, the second verification result, and the third verification result, generate the final verification result of the original text data.
[0110] The final verification result includes error type, error location, error cause, and correction suggestions. The final verification result is a structured report.
[0111] Optionally, in another embodiment of this application, the specific implementation of step S105 includes processes E1 to E4.
[0112] E1: Filter out the error locations and error types that failed the verification from the first verification result, the second verification result, and the third verification result.
[0113] The first, second, and third verification results each correspond to an error type.
[0114] Specifically, the error types include compliance errors, consistency errors, and logical errors. The first verification result corresponds to a compliance error, the second verification result corresponds to a consistency error, and the third verification result corresponds to a logical error.
[0115] It should be noted that each error type can be further subdivided into specific subtypes. For example, compliance errors include clause violations, missing keywords, etc.
[0116] E2: Input the correct criteria and the error content contained in the error location into the large model to obtain the cause of the error.
[0117] Understandably, the correct standards and the erroneous content contained in the erroneous locations are input into the large model. The large model then compares and analyzes the erroneous content with the correct standards (including compliance clauses, related data, and logical rules) to generate an explanation of the cause of the error.
[0118] For example, the statement "the credit card overdraft interest rate is 36%" in the text violates the requirement of "the overdraft interest rate cap is 24%" in the "XX" clause.
[0119] It should be noted that the text location capabilities of large models can also be used to accurately mark the start and end positions of errors in the original text (such as "line 5 of paragraph 3 to line 8 of paragraph 3").
[0120] E3: Input the error type and error cause into the large model to obtain correction suggestions.
[0121] Understandably, the error type and cause are input into a large model, which then leverages its text generation capabilities to generate targeted corrective suggestions based on the error cause and type. For example, the corrective suggestion might be to adjust the overdraft interest rate to no more than 24%.
[0122] E4: Generates the final verification result of the original text data based on the error location, error cause, error type, and correction suggestion information.
[0123] Optionally, in another embodiment of this application, the specific implementation of the large model is adjusted, including processes F1 to F4.
[0124] F1: Obtain a pre-trained large model and fine-tuned text data.
[0125] The fine-tuning text data includes correct text data and incorrect text data; the incorrect text data includes error types; the error types include at least compliance errors, consistency errors, and logical errors.
[0126] Optionally, the pre-trained large model should be a pre-trained large model with strong semantic understanding and logical reasoning capabilities, such as GPT-4, LLaMA3, etc.
[0127] F2: Annotate the fine-tuned text data to obtain the fine-tuned dataset.
[0128] It should be noted that the fine-tuning text data containing error types should be labeled to clarify the error type, location, and cause, thereby constructing a domain-specific fine-tuning dataset. The error types should comprehensively cover various types of errors, including compliance, consistency, and logical errors.
[0129] F3: Based on the fine-tuning dataset, the pre-trained large model is adjusted using an efficient parameter fine-tuning method to obtain the adjusted large model.
[0130] Specifically, based on a domain-specific fine-tuning dataset, an efficient parameter fine-tuning method is employed to fine-tune the base model. Specifically, most of the original model parameters are frozen, and only the parameters of the newly added adapter layer are trained. Backpropagation is used to minimize the loss between predictions and annotations, thus obtaining a fine-tuned model adapted to the target domain. The fine-tuning process uses a learning rate of 1e-5, a batch size of 8, and 10 training epochs to ensure stable model convergence.
[0131] F4: Adjust the parameters of the large model according to the current verification requirements to obtain the large model.
[0132] The current verification requirements include, but are not limited to, verification speed and verification accuracy.
[0133] It's important to note that, to ensure flexible adaptation to the speed and accuracy requirements of different validation tasks, we load the finely tuned large model into the validation system and encapsulate it as a service via an API interface. When invoked, generation parameters can be dynamically configured according to the current task requirements, such as the maximum generation length (default 512 tokens), temperature coefficient (default 0.7, used to adjust output randomness), and Top-P value (default 0.9, used to control candidate word diversity). This service receives output from the data preprocessing module, thereby providing accurate model inference support for the hierarchical validation module.
[0134] As can be seen, by employing an efficient parameter fine-tuning method, the basic large model can flexibly adapt to the text verification needs of different domains. This method replaces the traditional, tedious manual feature engineering, thereby significantly improving the model's generalization ability.
[0135] In addition, user feedback on the verification results is collected, including confirmation of correctness and correction data and correct annotations for errors such as misjudgments and omissions. This correction data is then integrated into the existing domain-specific fine-tuning dataset to construct an enhanced optimization dataset. Furthermore, based on this dataset, the current fine-tuned model is further fine-tuned by updating the adapter layer parameters, continuously improving the model's ability to identify complex error patterns. Finally, the iteratively updated model is version-managed, including tagging, saving, and supporting backtracking and switching, thereby ensuring the stability and reliability of the verification system during its continuous evolution.
[0136] It should be noted that, based on the processes shown in S101-S105 above, this embodiment can achieve the following beneficial effects:
[0137] 1. A large-scale model-based solution replaces traditional machine learning methods. Through large-scale pre-training, the large-scale model possesses powerful semantic understanding and logical reasoning capabilities, automatically capturing deep semantic features of text without relying on manually designed features. This effectively identifies implicit semantic errors, such as logical contradictions and semantic ambiguities, that are difficult to detect using traditional techniques. Simultaneously, combined with a layered verification strategy, compliance, consistency, and logical errors are systematically detected sequentially, achieving full coverage of both explicit and implicit errors and significantly improving overall verification accuracy.
[0138] 2. Achieving rapid domain adaptation of the basic large model through efficient parameter fine-tuning methods. For different domains, model customization can be completed simply by constructing the corresponding fine-tuning dataset, without redesigning features or training entirely new models for each domain. This allows the same solution to quickly adapt to text verification scenarios in multiple fields such as finance, healthcare, and law. Compared to existing technologies that require repeatedly developing independent models for different domains, the embodiments of this application greatly improve the model's generalization ability and effectively reduce the development costs of cross-domain applications.
[0139] 3. The validation logic is not based on fixed, manually applied rules, but rather relies on the large model's semantic understanding of domain terms and logical rules. When external validation requirements change, only the domain fine-tuning data needs to be updated and the model undergoes a lightweight secondary fine-tuning to respond quickly, without the need for manual redesign and debugging of complex rules. Furthermore, the built-in closed-loop iterative optimization mechanism continuously utilizes user feedback data to automatically optimize the model, significantly reducing manual maintenance costs and enabling it to flexibly adapt to continuously changing validation environments.
[0140] 4. Leveraging the large model's inherent text generation capabilities, the verification results are transformed into a structured report containing error type, specific location, cause analysis, and corrective suggestions. This fundamentally solves the deficiency of existing technologies, which typically only provide a binary "pass / fail" judgment without offering specific error information. Users can quickly locate and correct problems based on the clear report, greatly enhancing the practical value and operational efficiency of the verification results.
[0141] 5. This application embodiment constructs a fully automated system covering "data preprocessing - model adaptation - hierarchical verification - result interpretation - iterative optimization," realizing end-to-end automated processing from text input to report output, without any manual intervention. Compared with existing technical solutions that heavily rely on manual participation in feature engineering and rule adjustment, the automation level of this application embodiment is qualitatively improved, significantly reducing labor costs while ensuring efficient and stable verification output.
[0142] like Figure 2 The diagram shown is an architectural schematic of a text data verification device provided in an embodiment of this application. The verification device includes: an extraction unit 100, a first verification unit 200, a second verification unit 300, a third verification unit 400, and a generation unit 500.
[0143] Extraction unit 100 is used to extract the context information of each sentence in the original text data, and to construct text association data based on the context information of each sentence and the original text data.
[0144] The first verification unit 200 is used to verify the original text data using the compliance clauses of the domain in which the original text data is located, and to obtain the first verification result.
[0145] The first verification unit 200 is specifically used for: inputting the compliance clauses of the domain in which the original text data is located and the original text data into the large model to obtain the semantic similarity score between the original text data and the compliance clauses; determining whether the semantic similarity score is greater than the similarity score threshold; if the semantic similarity score is greater than the similarity score threshold, determining that the original text data passes the compliance verification and generating the first verification result; if the semantic similarity score is not greater than the similarity score threshold, determining that the original text data fails the compliance verification; marking the non-compliant content that fails the compliance verification in the original text data and associating the non-compliant content with the corresponding compliance clauses; generating the first verification result based on the conclusion that the original text data fails the compliance verification, the non-compliant content, and the corresponding compliance clauses.
[0146] The second verification unit 300 is used to verify the original text data using the associated reference data to obtain a second verification result.
[0147] The second verification unit 300 is specifically used for: inputting the original text data into the large model to obtain the key entities in the original text data; comparing the key entities with the entities in the associated reference data; if the key entities are consistent with the entities in the associated reference data, then the original text data is determined to have passed the consistency verification, and a second verification result is generated; if the key entities are inconsistent with the entities in the associated reference data, then the original text data is determined to have failed the consistency verification; marking the error locations and conflicting key entities that failed the consistency verification in the original text data; and generating a second verification result based on the conclusion that the original text data failed the consistency verification, the error locations, and the conflicting key entities.
[0148] The third verification unit 400 is used to perform text logic verification on the text-related data and obtain the third verification result.
[0149] The third verification unit 400 is specifically used for: inputting text-related data into the large model to obtain logical verification results; if the logical verification results indicate that the text-related data has logical contradictions or semantic ambiguities, then determining that the text-related data has failed the logical verification; marking the logical error locations and logically contradictory texts in the text-related data; generating a third verification result based on the conclusion that the text-related data has failed the logical verification, the logical error locations, and the logically contradictory texts; if the text logical verification results indicate that the text-related data does not have logical contradictions or semantic ambiguities, then determining that the text-related data has passed the logical verification and generating a third verification result.
[0150] The generation unit 500 is used to generate the final verification result of the original text data based on the first verification result, the second verification result, and the third verification result.
[0151] The generation unit 500 is specifically used for: filtering out the error locations and error types that failed the verification from the first verification result, the second verification result, and the third verification result; each of the first verification result, the second verification result, and the third verification result corresponds to an error type; inputting the correct standard and the error content contained in the error location into the large model to obtain the error cause; inputting the error type and error cause into the large model to obtain correction suggestion information; and generating the final verification result of the original text data based on the error location, error cause, error type, and correction suggestion information.
[0152] In summary, by combining a layered verification strategy to systematically detect compliance, consistency, and logical errors in sequence, both explicit and implicit errors are fully covered, significantly improving the overall verification accuracy.
[0153] Combination Figure 2 The verification device also includes, as shown, the following:
[0154] The acquisition unit is used to acquire a pre-trained large model and fine-tuned text data; the fine-tuned text data includes correct text data and incorrect text data; the incorrect text data includes error types; the error types include at least compliance errors, consistency errors, and logical errors.
[0155] The annotation unit is used to annotate the fine-tuned text data to obtain the fine-tuned dataset.
[0156] The tuning unit is used to tune a pre-trained large model based on a fine-tuning dataset using an efficient parameter fine-tuning method, resulting in a tuned large model.
[0157] The parameter adjustment unit is used to adjust the parameters of the large model according to the current verification requirements to obtain the large model.
[0158] Combination Figure 2 The verification device also includes, as shown, the following:
[0159] The cleaning unit is used to clean the text data to be processed, and obtain the cleaned text data.
[0160] The processing unit is used to standardize the cleaned text data to obtain the original text data.
[0161] Another embodiment of this application provides an electronic device, such as... Figure 3 As shown, it includes: memory 302, processor 301 and bus 303.
[0162] The memory 302 is used to store the program.
[0163] The processor 301 is used to execute a program, which, when executed, is specifically used to implement a text data verification method as provided in any of the above embodiments.
[0164] The electronic devices mentioned in this article can be servers, PCs, tablets, mobile phones, ECUs (Electronic Control Units), VCUs (Vehicle Control Units), MCUs (Micro Controller Units), HCUs (Hybrid Control Units), etc.
[0165] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed, implements a text data verification method as provided in any of the above embodiments.
[0166] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0167] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. In particular, for system or system embodiments, since they are fundamentally similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0168] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A text data verification method, characterized in that, include: Extract the context information of each sentence from the original text data, and construct text association data based on the context information of each sentence and the original text data; The original text data is validated using the compliance provisions of the domain in which it is located, resulting in a first validation result; The original text data is validated using associated reference data to obtain a second validation result; Perform text logic verification on the text-related data to obtain a third verification result; Based on the first verification result, the second verification result, and the third verification result, the final verification result of the original text data is generated.
2. The method according to claim 1, characterized in that, The first verification result is obtained by verifying the original text data using the compliance terms of the domain in which the original text data is located, including: The compliance clauses of the domain in which the original text data is located and the original text data are input into the large model to obtain the semantic similarity score between the original text data and the compliance clauses. Determine whether the semantic similarity score is greater than the similarity score threshold; If the semantic similarity score is greater than the similarity score threshold, then the original text data is determined to have passed the compliance verification, and a first verification result is generated; If the semantic similarity score is not greater than the similarity score threshold, then the original text data is determined to have failed the compliance verification. Mark the non-compliant content that fails the compliance check in the original text data and associate the non-compliant content with the corresponding compliance clauses; Based on the conclusion that the original text data failed the compliance check, the violation content, and the corresponding compliance clauses, a first verification result is generated.
3. The method according to claim 1, characterized in that, The step of verifying the original text data using associated reference data to obtain a second verification result includes: The original text data is input into the large model to obtain the key entities in the original text data; The key entities are compared with the entities in the associated reference data; If the key entity is consistent with the entity in the associated reference data, then the original text data is determined to have passed the consistency check, and a second check result is generated. If the key entity is inconsistent with the entity in the associated reference data, then the original text data is determined to have failed the consistency check. In the original text data, mark the error locations and conflicting key entities that failed the consistency check; A second verification result is generated based on the conclusion that the original text data failed the consistency check, the error location, and the conflicting key entity.
4. The method according to claim 1, characterized in that, The step of performing text logic verification on the text-related data to obtain a third verification result includes: The text association data is input into the large model to obtain the logical verification results; If the logical verification result indicates that the text association data has a logical contradiction or semantic ambiguity, then it is determined that the text association data has failed the logical verification. Mark the locations of logical errors and logically contradictory text in the text association data; A third verification result is generated based on the conclusion that the text association data failed the logical verification, the location of the logical error, and the text with logical contradictions. If the text logic verification result indicates that the text association data does not have logical contradictions or semantic ambiguities, then the text association data is determined to have passed the logic verification, and a third verification result is generated.
5. The method according to claim 1, characterized in that, The step of generating the final verification result of the original text data based on the first verification result, the second verification result, and the third verification result includes: The error locations and error types that failed the verification are filtered out from the first verification result, the second verification result, and the third verification result; each of the first verification result, the second verification result, and the third verification result corresponds to an error type. Input the correct criteria and the error content contained in the error location into the large model to obtain the cause of the error; Input the error type and error cause into the large model to obtain correction suggestion information; The final verification result of the original text data is generated based on the error location, the error cause, the error type, and the correction suggestion information.
6. The method according to any one of claims 2 to 5, characterized in that, The adjustment process of the large model includes: Acquire a pre-trained large model and fine-tuned text data; the fine-tuned text data includes correct text data and incorrect text data; the incorrect text data includes error types; the error types include at least compliance errors, consistency errors, and logical errors; The fine-tuned text data is labeled to obtain a fine-tuned dataset; Based on the fine-tuning dataset, the pre-trained large model is adjusted using a parameter-efficient fine-tuning method to obtain the adjusted large model. Adjust the parameters of the adjusted large model according to the current verification requirements to obtain the large model.
7. The method according to claim 1, characterized in that, Before extracting the context information of each sentence from the original text data, the process also includes: The text data to be processed is cleaned to obtain cleaned text data. The cleaned text data is then standardized to obtain the original text data.
8. A text data verification device, characterized in that, include: An extraction unit is used to extract the context information of each sentence in the original text data, and to construct text association data based on the context information of each sentence and the original text data; The first verification unit is used to verify the original text data using the compliance clauses of the domain in which the original text data is located, and to obtain a first verification result; The second verification unit is used to verify the original text data using associated reference data to obtain a second verification result; The third verification unit is used to perform text logic verification on the text-related data and obtain the third verification result. The generation unit is used to generate the final verification result of the original text data based on the first verification result, the second verification result, and the third verification result.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the text data verification method according to any one of claims 1-7.
10. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory are connected via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the text data verification method according to any one of claims 1-7.