Large language model JSON output repairing method and system based on ANTLR grammar analysis

Abstract

The invention provides a large language model JSON output repairing method and system based on ANTLR grammar analysis, and belongs to the technical field of model repairing. The method comprises the following steps: preprocessing an original output text of a model to obtain a to-be-repaired JSON text; constructing a fault-tolerant grammar rule, and generating a JSON grammar parser according to the fault-tolerant grammar rule; a JSON text to be repaired is input into an analyzer, grammar analysis is carried out in stages by adopting a progressive grammar matching strategy, and error node position coordinates and error types of grammar analysis anomalies are positioned by analyzing a context tracking mechanism; according to the position coordinates of the error nodes and the error types, performing context-aware repair on the text by adopting a hierarchical repair strategy to obtain a repaired JSON (JavaScript Object Notation) text; and performing multi-layer verification on the repaired JSON text by using a multi-layer verification mechanism, and outputting the repaired JSON text after the multi-layer verification is passed. According to the method, the error classification accuracy and the error positioning precision in the JSON output repairing process are improved, and then the repairing success rate of large language model JSON output is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of model repair, and in particular to a method and system for repairing JSON output of a large language model based on ANTLR grammar parsing. Background Art

[0002] Currently, large language models (LLMs) such as GPT and DeepSeek generally use the following processing methods when generating structured JSON data: (1) Direct text generation mode: The model directly generates JSON format text based on training data, relying on the model's own format constraint capabilities.

[0003] (2) Post-processing verification mode: Use a standard JSON syntax parser (such as Python's json.loads()) to verify the generated results, and regenerate or report an error if the parsing fails.

[0004] (3) Simple character repair mode: Use regular expressions or string replacement methods to repair obvious formatting errors, such as adding missing quotes, brackets, etc.

[0005] (4) Template constraint generation: Use prompt engineering or output format constraints to guide the model to generate the expected JSON structure.

[0006] However, the current large language models have the following technical flaws when generating structured JSON data: (1) Insufficient parsing error handling capabilities. When the standard JSON syntax parser encounters a format error, it directly throws an exception and cannot provide the specific error location and repair suggestions. The regeneration method is costly and cannot guarantee the semantic consistency after the repair.

[0007] (2) The repair strategy lacks structured understanding. The model cannot understand the nested structure and grammatical rules of JSON for simple string matching, which easily leads to problems such as over-repair or improper repair, such as adding punctuation marks in the wrong place.

[0008] (3) The error type recognition accuracy is low, and it is unable to accurately distinguish different error types such as grammatical errors, structural errors, and type errors. It also lacks the ability to analyze the propagation of errors in complex nested structures.

[0009] (4) The repair success rate and reliability are not high. The existing methods have a low repair success rate when processing complex JSON structures and lack a semantic verification mechanism for the repair results. Summary of the Invention

[0010] In view of this, the present invention provides a large language model JSON output repair method and system based on ANTLR syntax parsing to solve the problems of low error type recognition accuracy and low repair accuracy of current large language model output repair methods.

[0011] The technical solution adopted in the present invention is: In a first aspect, the present invention provides a method for repairing JSON output of a large language model based on ANTLR grammar parsing, comprising: Obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; Construct fault-tolerant grammar rules and generate a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules; Input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; According to the error node location coordinates and error type, a hierarchical repair strategy is used to perform context-aware repair on the JSON text to be repaired, and the repaired JSON text is obtained. The repaired JSON text is verified by using a multi-layer verification mechanism, and the repaired JSON text is output after the multi-layer verification passes.

[0012] Furthermore, the process of obtaining the original output text of the large language model and performing text preprocessing on the original output file to obtain the JSON text to be repaired includes: Get the original output text of the large language model and split the original output file by line to obtain multiple lines of original output text; Identify code block markers for multiple lines of raw output text, and use regular expressions to extract potential JSON fragments from multiple lines of raw output text based on the identified code block markers; Use the boundary recognition algorithm to identify the JSON boundaries in the potential JSON fragment, maintain a bracket stack, record the matching status of the boundary brackets, and mark it as a complete JSON candidate fragment when the bracket stack is empty; Perform character encoding standardization on the complete JSON candidate segment to obtain the JSON text to be repaired.

[0013] Furthermore, the construction of fault-tolerant grammar rules and the generation of a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules include: Define a two-layer grammar rule system including a standard JSON grammar rule layer and a fault-tolerant JSON grammar rule layer; the standard JSON grammar rule layer includes a set of standard grammar rules that conform to the JSON BNF specification, and the fault-tolerant JSON grammar rule layer includes an extended set of fault-tolerant grammar rules for format anomalies; Calculate the matching priority P_STD of the standard grammar rule and the matching priority P_TOL of the fault-tolerant grammar rule; The priority matching strategy of syntax parsing is set according to the matching priority P_STD and the matching priority P_TOL; wherein, the priority matching strategy of syntax parsing is specifically as follows: first, standard syntax matching is performed through the standard JSON syntax rule layer; if the standard syntax matching is successful, the standard syntax matching result is directly returned; if the standard syntax matching fails, the fault-tolerant JSON syntax rule layer is enabled to perform fault-tolerant syntax matching, and when the absolute value of the difference between the parsing success rate of the standard JSON syntax rule layer and the standard JSON syntax rule layer is less than half of the average value of the parsing accuracy of the standard JSON syntax rule layer and the standard JSON syntax rule layer, the fault-tolerant syntax matching result is returned; The two-layer grammar rule system and the priority matching strategy of grammar parsing are input into the ANTLR tool to generate a JSON grammar parser.

[0014] Furthermore, the grammar parsing is performed in stages using a progressive grammar matching strategy, including: Define the three syntax matching stages of the JSON syntax parser and set the matching success conditions and stage transition conditions for each syntax matching stage; the three syntax matching stages are strict matching stage S1, fault-tolerant matching stage S2, and fallback matching stage S3; The matching success conditions of each grammar matching stage are as follows: in S1 stage, the grammar rules are fully matched without any format deviation; in S2 stage, the grammar rules are partially matched, but there are tolerable format anomalies; in S3 stage, even if there are serious grammatical errors, the maximum structural information is retained; The specific phase transition conditions are as follows: when standard syntax parsing fails and the error type falls within the fault-tolerant range, the S1 phase transitions to the S2 phase; when fault-tolerant syntax parsing fails but some structural information can still be extracted, the S2 phase transitions to the S3 phase; According to the matching success conditions and stage transition conditions of the syntax matching stage, the JSON text to be repaired is progressively parsed in stages through the JSON syntax parser.

[0015] Furthermore, the establishment of a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism includes: Build a parsing context state machine and use it to analyze the error context of the abstract syntax tree nodes of the JSON parser during the parsing process; When the JSON parser encounters a syntax exception, the position coordinates and error type of the corresponding syntax exception error node are calculated; An error severity assessment is performed based on the error context of the syntax exception error node, and the error severity level is determined according to the error severity assessment result.

[0016] Furthermore, the layered repair strategy includes grammatical repair, structural repair, and semantic repair. The layered repair strategy is used to perform context-aware repair on the JSON text to be repaired based on the error node position coordinates and the error type to obtain the repaired JSON text, including: Locate the text location where the syntax error occurs in the JSON text to be repaired based on the error node location coordinates; Based on the context-aware repair algorithm, the repair order is determined according to the error type and the corresponding error severity level, and the corresponding hierarchical strategy is selected to build a repair plan. According to the repair plan, the syntactical errors of the JSON text to be repaired are repaired in a context-aware manner, and the repaired JSON text is output; When the current repair plan fails to repair the syntax error, the reason for the repair failure is recorded and the system is rolled back to the state before the repair. A new repair plan is selected according to the repair sequence to repair the syntax error until the syntax error is successfully repaired. Calculate the confidence score of each repair solution, and select the repair text corresponding to the repair solution with the highest confidence score as the repaired JSON text.

[0017] In a second aspect, the present invention provides a large language model JSON output repair system based on ANTLR grammar parsing, comprising: The text preprocessing module is used to obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; The engine builds modules, constructs fault-tolerant grammar rules, and generates a JSON grammar parser through the ANTLR tool based on the fault-tolerant grammar rules; The error diagnosis module is used to input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; The intelligent repair module is used to perform context-aware repair on the JSON text to be repaired based on the error node location coordinates and error type using a layered repair strategy to obtain the repaired JSON text; The verification module is used to perform multi-layer verification on the repaired JSON text using a multi-layer verification mechanism, and output the repaired JSON text after the multi-layer verification passes.

[0018] In summary, the beneficial effects of the present invention are as follows: The large language model JSON output repair method based on ANTLR grammar parsing provided by the present invention constructs fault-tolerant grammar rules and generates a JSON grammar parser through the ANTLR tool based on the fault-tolerant grammar rules. Under the premise of ensuring full compatibility with standard JSON, the success rate of parsing abnormal JSON is significantly improved. The method also inputs the JSON text to be repaired into the JSON grammar parser, adopts a progressive grammar matching strategy to perform grammar parsing in stages, and establishes a parsing context tracking mechanism based on a finite state automaton. The parsing context tracking mechanism locates the error node position coordinates and error type of the grammar parsing anomaly, solves the problem of rough error information and inaccurate positioning of traditional parsers, and improves the error classification accuracy and error positioning accuracy. At the same time, through the strategy of progressive degradation of grammar parsing, strong fault tolerance is provided while ensuring high-quality parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0020] Figure 1 This is a flow chart of the large language model JSON output repair method based on ANTLR grammar parsing of the present invention; Figure 2 This is a flowchart of the grammatical anomaly analysis of the present invention; Figure 3 It is a classification tree diagram of error types of the present invention; Figure 4 is a repair strategy decision flow chart of the present invention; Figure 5 This is a functional module block diagram of the large language model JSON output repair system based on ANTLR grammar parsing of the present invention; Figure 6 This is a functional workflow diagram of the large language model JSON output repair system based on ANTLR grammar parsing of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other and are all within the scope of protection of the present invention.

[0022] The detailed implementation process of the present invention is shown in the following examples.

[0023] Example 1: Reference Figure 1 As shown, Figure 1 This is a flow chart of the method for repairing the large language model JSON output based on ANTLR grammar parsing of the present invention. Figure 1 As shown, the method of the embodiment of the present invention includes: Obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; Construct fault-tolerant grammar rules and generate a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules; Input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; According to the error node location coordinates and error type, a hierarchical repair strategy is used to perform context-aware repair on the JSON text to be repaired, and the repaired JSON text is obtained. The repaired JSON text is verified by using a multi-layer verification mechanism, and the repaired JSON text is output after the multi-layer verification passes.

[0024] In this embodiment, the original output text of the large language model is obtained and the original output file is preprocessed to obtain the JSON text to be repaired. The specific processing includes the following steps: Get the raw output text of the large language model and split the raw output file by line to obtain multiple lines of raw output text. The raw output text of the large language model includes GPT output text, Deepseek output text, other LLM output text, and abnormal JSON text.

[0025] Identify code block tags for multiple lines of raw output text, and use regular expressions based on the identified code block tags (such as JSON) Extracts potential JSON fragments from multiple lines of raw output text.

[0026] A boundary identification algorithm is used to identify JSON boundaries in potential JSON fragments. A bracket stack is maintained to record the matching status of boundary brackets. When the bracket stack is empty, it is marked as a complete JSON candidate fragment. The boundary identification algorithm determines the start and end positions of JSON by counting bracket pairings.

[0027] Perform character encoding standardization on the complete JSON candidate fragment to obtain the JSON text to be repaired. Specifically, this process first removes unstructured text before and after the JSON, and then standardizes the quote type and encoding format. For example, non-standard quotes (such as Chinese quotes "") are converted to standard JSON quotes.

[0028] In this embodiment, fault-tolerant grammar rules are constructed, and based on the fault-tolerant grammar rules, a JSON grammar parser is generated by the ANTLR tool, which specifically includes the following processing steps: A two-layer grammar rule system is defined, which includes a standard JSON grammar rule layer and a fault-tolerant JSON grammar rule layer. The standard JSON grammar rule layer includes a set of standard grammar rules that conform to the JSON BNF specification, and the fault-tolerant JSON grammar rule layer includes an extended set of fault-tolerant grammar rules for format anomalies.

[0029] Calculate the matching priority P_STD of the standard grammar rule and the matching priority P_TOL of the error-tolerant grammar rule.

[0030] Set the priority matching strategy for syntax parsing based on the matching priority P_STD and matching priority P_TOL.

[0031] The grammatical rule R_i of this embodiment has a hierarchical priority match if and only if: P_STD>P_TOL ∧ Match_STD = false → Match_TOL = true ∧ |Success_STD- Success_TOL|<1 / 2(Precision_STD + Precision_TOL) Among them, P_STD represents the priority weight of the standard grammar rule; P_TOL represents the priority weight of the fault-tolerant grammar rule; Match_STD represents the standard grammar matching result; Match_TOL represents the fault-tolerant grammar matching result; Success_STD represents the standard grammar parsing success rate; Success_TOL represents the fault-tolerant grammar parsing success rate; Precision_STD represents the standard grammar parsing accuracy; Precision_TOL represents the fault-tolerant grammar parsing accuracy.

[0032] Therefore, the priority matching strategy of the syntax parsing of this embodiment is specifically as follows: first perform standard syntax matching through the standard JSON syntax rule layer. If the standard syntax matching is successful, the standard syntax matching result is directly returned; if the standard syntax matching fails, the fault-tolerant JSON syntax rule layer is enabled to perform fault-tolerant syntax matching, and when the absolute value of the difference between the parsing success rate of the standard JSON syntax rule layer and the standard JSON syntax rule layer is less than half of the average parsing accuracy of the standard JSON syntax rule layer and the standard JSON syntax rule layer, the fault-tolerant syntax matching result is returned.

[0033] The two-layer grammar rule system and the priority matching strategy of grammar parsing are input into the ANTLR tool to generate a JSON grammar parser.

[0034] This embodiment first establishes a two-layer grammar rule system. The standard layer adopts strict JSON BNF specifications, and the fault-tolerant layer designs special grammar rules for common abnormal patterns output by large language models. The parser adopts a priority-driven matching strategy, giving priority to using standard grammar for parsing. When the standard grammar fails to match, it automatically switches to fault-tolerant grammar for secondary parsing. By dynamically calculating the success rate and accuracy difference of the two-layer grammar, it ensures that fault-tolerant parsing maximizes the parsability of abnormal JSON while maintaining high accuracy. This method solves the "one-size-fits-all" problem of traditional JSON parsers. While ensuring full compatibility with standard JSON, it solves the problem that traditional JSON parsers cannot handle format anomalies. It can parse and locate various types of JSON syntax errors, significantly improving the parsing success rate of abnormal JSON from 0% to over 90%, while maintaining the 100% success rate of standard JSON parsing.

[0035] In some other embodiments of the present invention, a finite state automaton (FSA) may be used instead of ANTLR to parse JSON, and the specific implementation is as follows: (1) State definition: INIT → OBJECT_START → KEY → COLON → VALUE → COMMA → OBJECT_END; (2) Fault tolerance state: ERROR_RECOVERY → REPAIR_SUGGESTION → CONTINUE_PARSING.

[0036] The advantage of using finite state automata (FSA) is that it has faster parsing speed, less memory usage, and can adapt to real-time repair scenarios with extremely high performance requirements.

[0037] In some other embodiments, a sequence-to-sequence (Seq2Seq) model may be used to perform JSON repair, which is implemented as follows: (1) Training data: a large amount of paired data of incorrect JSON and correct JSON; (2) Model architecture: Transformer encoder-decoder structure; (3) Loss function: a composite loss function that combines grammatical loss and semantic loss; The advantage of using a sequence-to-sequence (Seq2Seq) model for JSON repair is that it can learn complex repair patterns and has strong adaptability.

[0038] In this embodiment, a progressive syntax matching strategy is used to perform syntax parsing in stages, including: Define the three syntax matching stages of the JSON syntax parser and set the matching success conditions and stage transition conditions for each syntax matching stage; the three syntax matching stages are strict matching stage S1, fault-tolerant matching stage S2, and fallback matching stage S3; The matching success conditions of each grammar matching stage are as follows: in S1 stage, the grammar rules are fully matched without any format deviation; in S2 stage, the grammar rules are partially matched, but there are tolerable format anomalies; in S3 stage, even if there are serious grammatical errors, the maximum structural information is retained; The specific phase transition conditions are as follows: when standard syntax parsing fails and the error type falls within the fault-tolerant range, the S1 phase transitions to the S2 phase; when fault-tolerant syntax parsing fails but some structural information can still be extracted, the S2 phase transitions to the S3 phase; According to the matching success conditions and stage transition conditions of the syntax matching stage, the JSON text to be repaired is progressively parsed in stages through the JSON syntax parser.

[0039] This embodiment designs a three-stage progressive parsing process, with grammatical rules of different degrees of strictness in each stage. The first stage uses standard JSON grammar to pursue complete format correctness; the second stage enables fault-tolerant grammatical rules, allowing common format deviations such as missing quotes, mismatched brackets, etc.; the third stage adopts the most relaxed fallback rules to retain useful structural information as much as possible even in the case of serious grammatical errors. Switching between stages is done through clear conversion conditions to ensure that information retention is maximized while obtaining the best parsing results. This method solves the problem of the parser's lack of flexibility when facing JSON with different degrees of abnormality. Through a progressive degradation strategy, it provides strong fault tolerance while ensuring high-quality parsing, so that the overall parsing success rate reaches 95%, while maintaining full compatibility with standard JSON.

[0040] In this embodiment, refer to Figure 2The parsing flowchart shown in the figure establishes a parsing context tracking mechanism based on a finite state automaton. The parsing context tracking mechanism locates the error node position coordinates and error type of the syntax parsing anomaly, including: Build a parsing context state machine and use it to analyze the error context of the abstract syntax tree nodes of the JSON parser during the parsing process; When the JSON parser encounters a syntax exception, the position coordinates and error type of the corresponding syntax exception error node are calculated; An error severity assessment is performed based on the error context of the syntax exception error node, and the error severity level is determined according to the error severity assessment result.

[0041] Specifically, in this embodiment, refer to Figure 3 As shown, error types include syntax errors, quote errors, delimiter errors, and data type errors. Syntax Errors include bracket mismatches, missing right curly brackets "}", missing left curly brackets "{", missing right square brackets "]", and missing left square brackets "[". Quote Errors include missing opening quotes, missing closing quotes, and quote type mismatches. Delimiter Errors include missing commas, extra commas, and incorrect delimiters (such as semicolons). Data type Errors include numeric format errors, Boolean value spelling errors, and null value errors.

[0042] Specifically, in some embodiments, the method flow of this embodiment for locating an erroneous node through context awareness is as follows: (1) Construct the parsing context state machine C_i. The state transition function of the state machine C_i is defined as follows: C_i = f(Token_current, AST_depth, Rule_stack, Error_history) Among them, Token_current represents the lexical unit currently being parsed; AST_depth represents the current depth of the abstract syntax tree; Rule_stack represents the state of the grammar rule stack; Error_history represents the historical error pattern record.

[0043] (2) Calculate the exact position coordinates (line, column, char_index) of the error node and the error type classification identifier E_type.

[0044] (3) Evaluate the error severity based on context information. The evaluation formula is as follows: Severity_score = α × Structure_impact + β × Semantic_consistency +γ × Recovery_difficulty Among them, structure_impact represents the impact of the error on the integrity of the JSON structure; semantic_consistency represents the impact of the error on semantic consistency; recovery_difficulty represents the difficulty coefficient of error repair; α, β, and γ are weight coefficients and α+β+γ=1.

[0045] This embodiment establishes a parsing context tracking mechanism based on a finite state automaton to record the state information of the parser in real time, including the currently processed lexical unit, AST (syntax tree) construction depth, grammar rule call stack, etc. When a parsing exception occurs, the state machine information is used to accurately locate the location of the error, providing not only row and column number information, but also the specific character index. At the same time, combined with historical error patterns, the error type is finely classified, and the error severity score is calculated to provide a quantitative basis for the subsequent selection of repair strategies. This method solves the problems of rough error information and inaccurate positioning of traditional parsers, improves the error positioning accuracy from the line level to the character level, and improves the error classification accuracy from 60% to 94%. It can identify 23 different types of JSON format exceptions.

[0046] In other embodiments of the present invention, the above-mentioned error diagnosis rules can be defined using a rule engine such as Drools. Example rules are: rule "missing right bracket detection" Rules defined in this way are highly readable and easy to maintain and extend.

[0047] In some other embodiments of the present invention, the JSON structure can be modeled as a graph and GNN can be used for error detection, which can be implemented as follows: (1) Node representation: JSON element (object, array, key-value pair); (2) Edge representation: JSON structure relationship (contains, adjacent); (3) GNN model: Use GraphSAGE or GAT for node classification.

[0048] This error diagnosis method can capture complex structural dependencies.

[0049] In this embodiment, the layered repair strategy includes syntax repair, structural repair and semantic repair. Among them, syntax repair is the first layer, which first analyzes the nesting level, then determines the insertion position, and finally performs error repair. The specific process of syntax repair is: First, the lexical analyzer scans the JSON text to identify matching states for grammatical symbols such as brackets and quotes. If a MISSING_RIGHT_BRACE error is detected, the analyze_brace_nesting method is called to analyze the nesting hierarchy of all left and right brackets from the beginning of the text to the error location.

[0050] Build a bracket stack structure, recording the position, type, and nesting depth of each unmatched left bracket.

[0051] The find_optimal_insertion_point method determines the optimal insertion point based on grammatical rules and contextual semantics. Insert the corresponding right bracket at the determined position to complete the grammar repair.

[0052] The syntax repair in this example accurately identifies and repairs over 90% of bracket mismatches, maintaining the integrity of the original data structure in the repaired JSON text. It also features fast processing speed, with a single bracket repair taking less than 10ms, and supports complex bracket matching repair for multi-layer nested structures.

[0053] The structural repair of this embodiment is the second layer, and its technical implementation process is as follows: Build an abstract syntax tree (AST) to analyze the structure and node relationships of JSON. When a missing comma error is detected, use the find_previous_complete_node method to locate the complete node before the error. Use the find_next_node_start method to identify the starting position of the next node.

[0054] The should_insert_comma method determines whether a comma should be inserted between two adjacent nodes, based on JSON syntax rules. The insert_comma_between_nodes method is called to insert a comma at the appropriate location, ensuring that the existing structure is not disrupted.

[0055] This example achieves over 95% structural repair accuracy, effectively resolving missing delimiters in arrays and objects, and can handle missing commas in complex nested structures. The repair process maintains the original order and hierarchical relationships of the data, supporting continuous repair of batch structural errors.

[0056] The semantic repair in this embodiment is the third layer, and its technical implementation process is as follows: Determine the expected data type for each field based on context inference and schema analysis. For string types, check whether the value is surrounded by quotes and automatically add double quotes if not.

[0057] For numeric types, fix: call the convert_to_number method to try to convert the string to a number, handling formats such as scientific notation and decimals.

[0058] For Boolean type fix: Use convert_to_boolean method to recognize "true", "false", "1", "0" and other values and convert them to standard Boolean format.

[0059] The semantic consistency of data is maintained during the repair process to avoid information loss.

[0060] This implementation achieves over 88% accuracy in data type repair, significantly improving the usability of JSON data. It intelligently identifies common type error patterns, such as numbers being incorrectly referenced as strings. The repaired data types conform to the JSON standard and support custom type conversion rules to accommodate diverse business scenarios.

[0061] The layered repair strategy of this embodiment solves the problems of lack of structured understanding and low repair accuracy in existing repair methods. By repairing at the three levels of syntax, structure, and semantics, the repair success rate is increased to 92%.

[0062] Reference Figure 4 As shown, based on the above-mentioned layered repair strategy, according to the error node position coordinates and error type, a layered repair strategy is adopted to perform context-aware repair on the JSON text to be repaired, and the repaired JSON text is obtained, which specifically includes: Locate the text location where the syntax error occurs in the JSON text to be repaired based on the error node location coordinates; Based on the context-aware repair algorithm, the repair order is determined according to the error type and the corresponding error severity level, and the corresponding hierarchical strategy is selected to build a repair plan. According to the repair plan, the syntactical errors of the JSON text to be repaired are repaired in a context-aware manner, and the repaired JSON text is output; When the current repair plan fails to repair the syntax error, the reason for the repair failure is recorded and the system is rolled back to the state before the repair. A new repair plan is selected according to the repair sequence to repair the syntax error until the syntax error is successfully repaired. Calculate the confidence score of each repair solution, and select the repair text corresponding to the repair solution with the highest confidence score as the repaired JSON text.

[0063] The context-aware repair algorithm in this embodiment primarily maintains the JSON Schema inference mechanism, inferring the expected structure based on the parsed portion. It also implements retrospective repair, falling back to try other strategies when one repair strategy fails, and selecting the optimal repair solution using confidence scores.

[0064] The implementation process of the context-aware repair algorithm technology in this embodiment is as follows: 1. Establish a Schema inference mechanism: Analyze the successfully parsed JSON fragments, extract information such as field names, data types, and nested structures, and build a dynamic Schema model.

[0065] 2. Set a retrospective repair strategy: When a repair plan fails, record the cause and location of the failure, roll back to the state before the repair, and try alternative repair plans according to priority.

[0066] 3. Confidence scoring algorithm: Calculate a confidence score for each repair solution, taking into account factors such as grammatical correctness (40%), structural integrity (35%), and semantic rationality (25%).

[0067] 4. Optimal solution selection: Select the repair solution with the highest confidence and passed verification as the final result.

[0068] Through the context-aware repair algorithm of this embodiment, the overall repair success rate of JSON abnormal text can be increased to more than 92%, which is 15% higher than that of a single strategy. The repair quality is significantly improved, and the secondary error rate is reduced by 80%. At the same time, it also supports intelligent decision-making in complex scenarios and adapts to diverse JSON error patterns. It has self-learning capabilities and can optimize strategy weights based on repair history to improve repair adaptability. The context-aware repair algorithm of this embodiment solves the problem that traditional error analysis methods cannot accurately identify error types and severity.

[0069] In other embodiments of the present invention, the layered repair strategy can establish a repair solution based on template matching by pre-defining repair templates of common error patterns. This repair method is fast and has predictable results. The template example is as follows: Template name: MISSING_COMMA_BETWEEN_OBJECTS; Error mode:}"key":value; Fix template:}, "key": value; Applicable conditions: The previous character is '}', and the following character is a key-value pair; In this embodiment, the multi-layer verification mechanism includes syntax verification, structural integrity verification, and semantic consistency verification.

[0070] Specifically, the syntax verification recursive descent parser algorithm (Recursive Descent Parser) is implemented. The specific technical implementation process is as follows: Finite state automaton (FSA) is used for lexical analysis to decompose JSON text into token sequences; Use recursive descent parser to build syntax tree according to JSON grammar rules; Implement bracket matching verification: Use the stack structure to verify the pairing integrity of symbols such as '{', '[', '(', etc. String format verification: Check the validity of quotation mark pairing and escape characters, and use the regular expression / "(\\.|[^"\\])*" / for pattern matching; Numeric format validation: Validates numeric formats based on the JSON specification, supporting integer, floating-point, scientific notation, and other formats.

[0071] This example achieves 99.8% syntax verification accuracy, identifying all standard JSON syntax errors. Verification is fast, with an average processing time of 0.5ms / KB. It also supports incremental verification, revalidating only the corrected parts and providing detailed error location and type information to facilitate subsequent repairs.

[0072] In this embodiment, structural integrity verification is implemented using a graph theory algorithm combined with abstract syntax tree (AST) analysis. The specific implementation process is as follows: 1. Build a directed acyclic graph (DAG) of JSON structure, where nodes represent data elements and edges represent inclusion relationships.

[0073] 2. Use the depth-first search (DFS) algorithm to traverse the entire structure tree and check the integrity of the nodes.

[0074] 3. Implement structural consistency checks, including: (1) Object structure verification: Ensure that the format of each key-value pair is correct and the key must be a string type; (2) Array structure verification: check the integrity of element separators and type consistency; (3) Nesting level verification: limit the maximum nesting depth to prevent stack overflow.

[0075] 4. Use topological sorting algorithm to detect circular reference problems.

[0076] 5. Apply structural hashing algorithm to calculate structural fingerprints and quickly compare structural differences.

[0077] The structural integrity verification accuracy of this embodiment reaches 96.5%, which can detect inconsistencies in complex nested structures. It also supports structural verification of large-scale JSON documents, with a processing capacity of 10MB / s. It also provides a structural difference report to accurately locate problem nodes.

[0078] In this embodiment, semantic consistency verification is implemented by using an improved edit distance algorithm combined with semantic similarity calculation. The specific implementation process is as follows: 1. Improved edit distance algorithm: First, the structural edit distance between the original JSON and the repaired JSON is calculated. Then, weights are assigned to different operations, specifically insertion (weight 0.8), deletion (weight 1.2), and replacement (weight 1.0). Finally, data type matching is considered, and the weight of same-type replacements is reduced by 50%.

[0079] 2. Set the semantic similarity calculation formula: Similarity = 1 - (Weighted_Edit_Distance / Max_Possible_Distance) Similarity represents similarity, which is the semantic similarity score between the original JSON and the repaired JSON. Its value range is [0, 1], and the corresponding range values have the following meanings: 1.0: completely similar, the repaired JSON has the same semantics as the original JSON; 0.8-0.99: Highly similar, the repair is reasonable and the original semantics are maintained; 0.6-0.79: Moderately similar, the repair is acceptable but there are some semantic changes; <0.6: low similarity, the restoration may have changed the original semantics; Weighted_Edit_Distance represents the weighted edit distance, which is the edit distance that takes the operation weight into account and represents the minimum weighted operation cost required to convert the original JSON into the repaired JSON.

[0080] The weighted edit distance is calculated as: Weighted_Edit_Distance = Σ(number of insert operations × 0.8 + number of delete operations × 1.2 + number of replace operations × 1.0) The weighted edit distance calculation weight design is as follows: Insertion has a weight of 0.8, indicating that the repair process adds missing content (such as brackets or commas) with minimal semantic impact. Deletion has a weight of 1.2, indicating that deleting existing content has a significant semantic impact and is the highest weight. Replacement has a weight of 1.0, indicating that existing content is modified, with a moderate semantic impact.

[0081] Max_Possible_Distance represents the maximum possible distance, that is, the maximum edit distance that can theoretically be achieved under the given JSON structure. Its calculation method is: Max_Possible_Distance = max(len(original_tokens), len(repaired_tokens)) × max_weight Where: original_tokens represents the length of the token sequence of the original JSON; repaired_tokens represents the length of the token sequence of the repaired JSON; max_weight represents the maximum weight value (1.2) among all operations.

[0082] Normalization is achieved by dividing by the maximum possible distance, making the similarity within the range of [0,1]. The normalization calculation logic is: when Weighted_Edit_Distance = 0, Similarity = 1 (identical). When Weighted_Edit_Distance = Max_Possible_Distance, Similarity = 0 (completely different).

[0083] Suppose there is the following repair case: Original JSON: {"name": "张三", "age": 25 (missing right parenthesis; Repaired JSON: {"name": "张三", "age": 25}.

[0084] The specific calculation process is: Perform 1 insertion operation (add "}"); Weighted_Edit_Distance = 1 × 0.8 = 0.8; Max_Possible_Distance = 6 × 1.2 = 7.2 (6 tokens, maximum weight 1.2); Similarity = 1 - (0.8 / 7.2) = 0.889; The similarity score of the above repair case is 0.889, indicating a high repair quality and good semantic preservation.

[0085] 3. Multi-dimensional semantic checking, including key name similarity, value range verification, and type consistency. Key name similarity uses the Jaccard similarity coefficient to calculate key set similarity. Value range verification checks whether the repaired value is within a reasonable range. Type consistency verifies whether the repaired data type meets the contextual expectations.

[0086] 4. Adaptive threshold mechanism, dynamically adjusts the minimum similarity threshold based on the complexity of JSON. The minimum similarity threshold for simple structures is min_similarity_threshold = 0.85, and the minimum similarity threshold for complex structures is min_similarity_threshold = 0.75.

[0087] The semantic consistency verification accuracy of this embodiment reaches 91.2%, which can effectively prevent semantic deviation caused by excessive repair. It also supports customized semantic rules, adapts to different business scenarios, has high computational efficiency, and an average processing time of 2ms / KB.

[0088] This embodiment utilizes the aforementioned layered verification mechanism to achieve a comprehensive three-tier verification accuracy of 94.8%. The total verification process time is kept within 5ms / KB, and parallel verification is supported, improving verification performance by 60% in multi-core environments. Furthermore, layered verification reports are provided to facilitate problem location and optimize remediation strategies.

[0089] In other embodiments of the present invention, temporal logic or first-order logic may be used to describe the JSON specification, and the repair result may be verified by model checking. The verification result is mathematically rigorous, and the implementation method is as follows: Examples of formal specifications are: ∀object ∈ JSON : isWellFormed(object) ↔ hasMatchingBraces(object) ∧ hasValidKeyValuePairs(object) ∧ hasCorrectSeparators(object) In addition, you can use the contrastive learning framework to train the JSON quality assessment model for verification. This method can learn subtle quality differences and achieve higher verification accuracy.

[0090] In addition, the repair quality can be independently evaluated from multiple dimensions, and then weighted and integrated to establish a verification scheme based on multi-dimensional scoring. The evaluation dimensions of this scheme are: syntax dimension score = JSON syntax correctness check result (0-1); structure dimension score = structure integrity assessment score (0-1); semantic dimension score = semantic similarity calculation result (0-1); type dimension score = data type consistency score (0-1).

[0091] The overall score = w1 × grammatical score + w2 × structural score + w3 × semantic score + w4 × type score. w1 + w2 + w3 + w4 = 1, and the weights can be adjusted based on the application scenario. Independently assessing repair quality across multiple dimensions provides a more comprehensive assessment, and the weights can be adjusted based on needs.

[0092] In this embodiment, the technical principle of the present invention is further explained by taking the problem of fixing the mismatched brackets of complex nested JSON as an example and combining some implementation codes. The specific implementation process is as follows: 1. Input abnormal JSON { "user": { "name": "Zhang San", "profile": { "age": 25, "skills": ["Python", "Java" } }, "status": "active" 2. Repair process: 1. Preprocessing stage: Extract the JSON fragment and identify the missing 2 closing brackets.

[0093] 2. ANTLR parsing phase: The fault-tolerant grammar rule matches the "skills" array and finds that the "skills" object lacks a right square bracket. It also recognizes that the "profile" object lacks a right curly bracket. 3. Error diagnosis stage: Error 1: Line 6, column 24, missing ']'; Error 2: Line 7, column 5, missing '}' 4. Intelligent Repair Phase: Analyze the nesting relationship: The skills array is nested within the profile object. Determine the repair order: repair the inner array first, then the outer object. Execute the repair: Add a ']' after "Java" and a '}' after line 7.

[0094] 5. Verification stage: Syntax verification passed, structural integrity verification passed, semantic consistency score: 0.98 (meets the threshold of 0.85).

[0095] 3. The repair results are: { "user": { "name": "Zhang San", "profile": { "age": 25, "skills": ["Python", "Java"] } }, "status": "active" } The embodiment of the present invention constructs fault-tolerant grammar rules and generates a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules. This significantly improves the success rate of parsing abnormal JSON while ensuring full compatibility with standard JSON. The method also inputs the JSON text to be repaired into the JSON grammar parser, uses a progressive grammar matching strategy to perform grammar parsing in stages, and establishes a parsing context tracking mechanism based on a finite state automaton. The parsing context tracking mechanism locates the position coordinates and error type of the error node of the grammar parsing anomaly, solving the problem of coarse error information and inaccurate positioning of traditional parsers, and improving the accuracy of error classification and error location. At the same time, through the strategy of progressive degradation of grammar parsing, strong fault tolerance is provided while ensuring high-quality parsing.

[0096] Example 2: Reference Figure 5 and Figure 6 As shown, based on Example 1, this embodiment further provides a large language model JSON output repair system based on ANTLR grammar parsing, the system comprising: The text preprocessing module is used to obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; The engine builds modules, constructs fault-tolerant grammar rules, and generates a JSON grammar parser through the ANTLR tool based on the fault-tolerant grammar rules; The error diagnosis module is used to input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; The intelligent repair module is used to perform context-aware repair on the JSON text to be repaired based on the error node location coordinates and error type using a layered repair strategy to obtain the repaired JSON text; The verification module is used to perform multi-layer verification on the repaired JSON text using a multi-layer verification mechanism, and output the repaired JSON text after the multi-layer verification passes.

[0097] Reference Figure 6 As shown, the core processing flow of this embodiment of the system is: 1. LLMs (Large Language Models) output → 2. Preprocessing → 3. ANTLR parsing → 4. Error diagnosis → 5. Intelligent repair → 6. Verification → 7. Result output. The specific working principle of the system includes the following steps: Preprocessing steps: split the large language model output text by line, identify code block tags and extract content, use a state machine algorithm to identify JSON boundaries, maintain a bracket stack to determine complete JSON candidate fragments, remove unstructured text before and after JSON, and standardize quotation mark types and encoding formats. ANTLR parsing step: Use enhanced fault-tolerant grammar rules, including tolerantObject and tolerantArray fault-tolerant grammar rules, to parse the candidate JSON fragment. Error diagnosis steps: Build a parsing context state machine, calculate the precise location coordinates of the error node and the error type classification identifier, and evaluate the error severity based on the context information. Intelligent repair steps: Adopt a layered repair strategy, first perform grammatical repair, then structural repair, and finally semantic repair. Verification steps: First perform syntax verification, then perform structural integrity verification, and finally perform semantic consistency verification.

[0098] This embodiment effectively extracts and preprocesses JSON fragments through the preprocessing module. The enhanced fault-tolerant grammar rules of the ANTLR parsing engine improve the ability to parse abnormal JSON. The error diagnosis module accurately locates, classifies, and evaluates errors. The layered repair strategy of the intelligent repair module improves the accuracy and completeness of repairs. The multi-layer verification mechanism of the verification module ensures the quality of the repaired JSON. Overall, this improves the success rate and reliability of repairing JSON output from large language models, resolving many issues existing in existing technologies.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for repairing JSON output of a large language model based on ANTLR grammar parsing, characterized in that: include: Obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; Construct fault-tolerant grammar rules and generate a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules; Input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; According to the error node location coordinates and error type, a hierarchical repair strategy is used to perform context-aware repair on the JSON text to be repaired, and the repaired JSON text is obtained. The repaired JSON text is verified by using a multi-layer verification mechanism, and the repaired JSON text is output after the multi-layer verification passes.

2. The large language model JSON output repair method based on ANTLR grammar parsing according to claim 1 is characterized in that, The method of obtaining the original output text of the large language model and performing text preprocessing on the original output file to obtain the JSON text to be repaired includes: Get the original output text of the large language model and split the original output file by line to obtain multiple lines of original output text; Identify code block markers for multiple lines of raw output text, and use regular expressions to extract potential JSON fragments from multiple lines of raw output text based on the identified code block markers; Use the boundary recognition algorithm to identify the JSON boundaries in the potential JSON fragment, maintain a bracket stack, record the matching status of the boundary brackets, and mark it as a complete JSON candidate fragment when the bracket stack is empty; Perform character encoding standardization on the complete JSON candidate segment to obtain the JSON text to be repaired.

3. The large language model JSON output repair method based on ANTLR grammar parsing according to claim 1 is characterized in that, The method of constructing fault-tolerant grammar rules and generating a JSON grammar parser using the ANTLR tool based on the fault-tolerant grammar rules includes: Define a two-layer grammar rule system including a standard JSON grammar rule layer and a fault-tolerant JSON grammar rule layer; the standard JSON grammar rule layer includes a set of standard grammar rules that conform to the JSON BNF specification, and the fault-tolerant JSON grammar rule layer includes an extended set of fault-tolerant grammar rules for format anomalies; Calculate the matching priority P_STD of the standard grammar rule and the matching priority P_TOL of the fault-tolerant grammar rule; The priority matching strategy of syntax parsing is set according to the matching priority P_STD and the matching priority P_TOL; wherein, the priority matching strategy of syntax parsing is specifically as follows: first, standard syntax matching is performed through the standard JSON syntax rule layer; if the standard syntax matching is successful, the standard syntax matching result is directly returned; if the standard syntax matching fails, the fault-tolerant JSON syntax rule layer is enabled to perform fault-tolerant syntax matching, and when the absolute value of the difference between the parsing success rate of the standard JSON syntax rule layer and the standard JSON syntax rule layer is less than half of the average value of the parsing accuracy of the standard JSON syntax rule layer and the standard JSON syntax rule layer, the fault-tolerant syntax matching result is returned; The two-layer grammar rule system and the priority matching strategy of grammar parsing are input into the ANTLR tool to generate a JSON grammar parser.

4. The large language model JSON output repair method based on ANTLR grammar parsing according to claim 1 is characterized in that, The progressive grammar matching strategy is used to perform grammar parsing in stages, including: Define the three syntax matching stages of the JSON syntax parser and set the matching success conditions and stage transition conditions for each syntax matching stage; the three syntax matching stages are strict matching stage S1, fault-tolerant matching stage S2, and fallback matching stage S3; The matching success conditions of each grammar matching stage are as follows: in S1 stage, the grammar rules are fully matched without any format deviation; in S2 stage, the grammar rules are partially matched, but there are tolerable format anomalies; in S3 stage, even if there are serious grammatical errors, the maximum structural information is retained; The specific phase transition conditions are as follows: when standard syntax parsing fails and the error type falls within the fault-tolerant range, the S1 phase transitions to the S2 phase; when fault-tolerant syntax parsing fails but some structural information can still be extracted, the S2 phase transitions to the S3 phase; According to the matching success conditions and stage transition conditions of the syntax matching stage, the JSON text to be repaired is progressively parsed in stages through the JSON syntax parser.

5. The large language model JSON output repair method based on ANTLR grammar parsing according to claim 1 is characterized in that, The establishment of a parsing context tracking mechanism based on a finite state automaton, and locating the error node position coordinates and error type of the syntax parsing exception through the parsing context tracking mechanism, includes: Build a parsing context state machine and use it to analyze the error context of the abstract syntax tree nodes of the JSON parser during the parsing process; When the JSON parser encounters a syntax exception, the position coordinates and error type of the corresponding syntax exception error node are calculated; An error severity assessment is performed based on the error context of the syntax exception error node, and the error severity level is determined according to the error severity assessment result.

6. The large language model JSON output repair method based on ANTLR grammar parsing according to claim 1 is characterized in that, The layered repair strategy includes grammatical repair, structural repair, and semantic repair. The layered repair strategy is used to perform context-aware repair on the JSON text to be repaired based on the error node position coordinates and error type to obtain the repaired JSON text, including: Locate the text location where the syntax error occurs in the JSON text to be repaired based on the error node location coordinates; Based on the context-aware repair algorithm, the repair order is determined according to the error type and the corresponding error severity level, and the corresponding hierarchical strategy is selected to build a repair plan. According to the repair plan, the syntactical errors of the JSON text to be repaired are repaired in a context-aware manner, and the repaired JSON text is output; When the current repair plan fails to repair the syntax error, the reason for the repair failure is recorded and the system is rolled back to the state before the repair. A new repair plan is selected according to the repair sequence to repair the syntax error until the syntax error is successfully repaired. Calculate the confidence score of each repair solution, and select the repair text corresponding to the repair solution with the highest confidence score as the repaired JSON text.

7. A large language model JSON output repair system based on ANTLR grammar parsing, which is implemented by the large language model JSON output repair method based on ANTLR grammar parsing according to any one of claims 1 to 6, characterized in that: include: The text preprocessing module is used to obtain the original output text of the large language model and perform text preprocessing on the original output file to obtain the JSON text to be repaired; The engine builds modules, constructs fault-tolerant grammar rules, and generates a JSON grammar parser through the ANTLR tool based on the fault-tolerant grammar rules; The error diagnosis module is used to input the JSON text to be repaired into the JSON syntax parser, use a progressive syntax matching strategy to perform syntax parsing in stages, and establish a parsing context tracking mechanism based on a finite state automaton to locate the error node position coordinates and error type of the syntax parsing anomaly through the parsing context tracking mechanism; The intelligent repair module is used to perform context-aware repair on the JSON text to be repaired based on the error node location coordinates and error type using a layered repair strategy to obtain the repaired JSON text; The verification module is used to perform multi-layer verification on the repaired JSON text using a multi-layer verification mechanism, and output the repaired JSON text after the multi-layer verification passes.

Images